文献速递：基于SAM的医学图像分割---UN-SAM: 通用无需提示的广义细胞核图像分割-编程知识

Title

题目

UN-SAM: Universal Prompt-Free Segmentation

for Generalized Nuclei Images

UN-SAM: 通用无需提示的广义细胞核图像分割

文献速递介绍

在数字病理学领域，对病理组织图像进行细胞核图像分割的任务是形态量化和肿瘤分级评估的基石。尽管这项任务非常重要，但由于细胞核高度密集的特性，它给细胞核图像分割任务带来了相当大的困难。

本项工作得到了香港研究资助局（RGC）通用研究基金14204321、14220622和InnoHK计划的支持。(等同贡献: Z. Chen 和 Q. Xu, 通讯作者: Yixuan Yuan)

特别是，为自然图像设计的实例分割方法依赖于区域提议，并且难以区分相邻的细胞核，从而妨碍了对单个细胞核的精确分割。在这一挑战性背景下，通过为细胞核的特性量身定制的网络结构和监督策略，研究了经典的细胞核图像分割算法。这些研究在同质数据集下的全监督条件下表现出高准确率，并在推进该领域发展中发挥了核心作用。然而，这些方法需要精心设计的后处理来识别细胞核实例，对密集的超参数搜索很敏感。更严重的是，当面对日常临床实践中遇到的组织类型、染色协议和成像条件的固有异质性时，它们的局限性变得明显。这种可变性对于开发具有必要通用性和准确性的分割模型across不同数据集提出了重大挑战。

最近，“分割任何物体模型”（SAM）的出现开启了分割的新时代，并在解决这些问题上显示出超越传统限制的多功能能力。SAM在自然图像场景中的显著效能已经得到了良好的验证，展示了它在多种场景中的鲁棒性和适应性。基于此，SAM开始渗透到医学成像社区，并展示了其在医学图像分割方面的能力，潜力适用于多种场景。此外，初步调查表明SAM能够适应医学数据提出的独特挑战，尽管对于临床部署还需要进一步完善。

Abstract

摘要

In digital pathology, precise nuclei segmen tation is pivotal yet challenged by the diversity of tis sue types, staining protocols, and imaging conditions. Re cently, the segment anything model (SAM) revealed over whelming performance in natural scenarios and impressive adaptation to medical imaging. Despite these advantages,the reliance of labor-intensive manual annotation as seg mentation prompts severely hinders their clinical applica bility, especially for nuclei image analysis containing mas sive cells where dense manual prompts are impractical. To overcome the limitations of current SAM methods while retaining the advantages, we propose the Universal prompt free SAM framework for Nuclei segmentation (UN-SAM), by providing a fully automated solution with remarkable generalization capabilities. Specifically, to eliminate the labor-intensive requirement of per-nuclei annotations for prompt, we devise a multi-scale Self-Prompt Generation (SPGen) module to revolutionize clinical workflow by au tomatically generating high-quality mask hints to guide the segmentation tasks. Moreover, to unleash the generaliza tion capability of SAM across a variety of nuclei images, we devise a Domain-adaptive Tuning Encoder (DT-Encoder) to seamlessly harmonize visual features with domain common and domain-specific knowledge, and further de vise a Domain Query-enhanced Decoder (DQ-Decoder) by leveraging learnable domain queries for segmentation decoding in different nuclei domains. Extensive experi ments prove that UN-SAM with exceptional performance surpasses state-of-the-arts in nuclei instance and seman tic segmentation, especially the generalization capability in zero-shot scenarios. The source code is available at https://github.com/CUHK-AIM-Group/UN-SAM. Index Terms**—** Nuclei image, semantic segmentation, in stance segmentation, domain generalization.

在数字病理学中，精确的细胞核分割至关重要，但由于组织类型、染色协议和成像条件的多样性，这一任务面临挑战。最近，“分割任何物体模型”（Segment Anything Model, SAM）在自然场景中展现出了卓越的性能，并在医学成像中表现出了令人印象深刻的适应性。尽管如此，依赖于费时的人工注释作为分割提示的方式严重阻碍了其在临床应用中的可行性，特别是在包含大量细胞的细胞核图像分析中，密集的手工提示是不切实际的。

为了克服当前SAM方法的限制，同时保留其优势，我们提出了一种用于细胞核分割的通用无提示SAM框架（UN-SAM），提供了一种具有显著泛化能力的全自动解决方案。具体来说，为了消除每个细胞核注释的手工劳动需求，我们设计了一个多尺度自我提示生成（Self-Prompt Generation, SPGen）模块，通过自动生成高质量的掩码提示来引导分割任务，从而革新了临床工作流程。此外，为了释放SAM在各种细胞核图像中的泛化能力，我们设计了一个域自适应调整编码器（Domain-adaptive Tuning Encoder, DT-Encoder），以无缝地协调视觉特征与域通用及域特定知识，并进一步设计了一个利用可学习的域查询进行分割解码的域查询增强解码器（Domain Query-enhanced Decoder, DQ-Decoder）。广泛的实验证明，UN-SAM在细胞核实例和语义分割中的表现超越了最先进的方法，特别是在零样本情况下的泛化能力。源代码可在https://github.com/CUHK-AIM-Group/UN-SAM 获取。

索引术语：细胞核图像，语义分割，实例分割，域泛化。

METHOD

方法

A. Overview of UN-SAM

As illustrated in Fig. 2, we present the UN-SAM to provide automatic segmentation with remarkable generalization across a variety of nuclei images. Given nuclei images from the k-th domain, we first utilize the Domain-adaptive Tuning Encoder (DT-Encoder) to generate domain knowledge-enhanced image embeddings. Then, these embeddings are delivered to the multi-scale Self-Prompt Generation (SPGen) module, which autonomously produces a set of self-generated prompt tokens to guide nuclei segmentation. Following this, the Domain Query-enhanced Decoder (DQ-Decoder) leverages the query embedding of the corresponding domain to accurately predict the segmentation mask. In general, this novel UN-SAM, comprising the DT-Encoder and DQ-Decoder, is specifically designed to tackle the chal lenges of model generalization across varying domains in nuclei image segmentation. Meanwhile, the SPGen module eliminates the demand of labor-intensive manual annotations for prompts that are typically associated with the SAM frame works . Therefore, these key modules form our UN SAM that enhances the generalization and applicability to a wide array of nuclei image segmentation.

正如图 2 所示，我们提出了UN-SAM，以提供自动化分割，并在多种细胞核图像中展现出显著的泛化能力。对于来自第k个领域的细胞核图像，我们首先使用领域自适应调整编码器（Domain-adaptive Tuning Encoder, DT-Encoder）生成增强了领域知识的图像嵌入。然后，这些嵌入被送入多尺度自我提示生成（Self-Prompt Generation, SPGen）模块，该模块自主产生一组自生成的提示标记以指导细胞核分割。接着，领域查询增强解码器（Domain Query-enhanced Decoder, DQ-Decoder）利用相应领域的查询嵌入来准确预测分割掩模。

总的来说，这个新颖的UN-SAM，包括DT-Encoder和DQ-Decoder，特别设计用于解决在细胞核图像分割中跨不同领域的模型泛化挑战。同时，SPGen模块消除了与SAM框架通常相关的、需要大量劳动的手工注释的需求。因此，这些关键模块构成了我们的UN-SAM，增强了对广泛细胞核图像分割的泛化能力和适用性。

CONCLUSION

结论

In this work, we propose the UN-SAM framework to

achieve a universal solution for nuclei segmentation within the realm of digital pathology. In particular, the DT-Encoder is proposed to harmonize domain-common and domain-specific features within the image encoder, which propels the gen eralization potential of SAM across diverse nuclei datasets.

Then, the SPGen module is devised to autonomously pro duce high-quality mask hints instead of manual annotated prompt, to facilitate the clinical workflow towards enhanced efficiency. Moreover, we further propose the DQ-Decoder to leverage learnable domain queries for segmentation decoding in different nuclei domains. Extensive experiments confirm the advantage of UN-SAM over existing medical SAMs by negating the need for labor-intensive manual annotations and offering an unparalleled zero-shot learning capability that adapts to varying domains with ease.

在这项工作中，我们提出了UN-SAM框架，旨在数字病理学领域内实现一个针对细胞核分割的通用解决方案。特别是，提出了DT-Encoder来协调图像编码器内的领域通用和领域特定特征，这推动了SAM在不同细胞核数据集上的泛化潜力。接着，设计了SPGen模块，自主产生高质量的掩模提示而不是手动注释的提示，以促进临床工作流程向提高效率的方向发展。此外，我们还提出了DQ-Decoder，利用可学习的领域查询来进行不同细胞核领域的分割解码。广泛的实验证实了UN-SAM相对于现有的医学SAMs的优势，它取消了对劳动密集型手工注释的需求，并提供了无与伦比的零样本学习能力，轻松适应不同领域。

Fig

图

Fig. 1. Performance comparison on nuclei image segmentation. The semantic segmentation is measured by Dice, and instance segmenta tion (marked as inst.) is measured by Aggregated Jaccard Index (AJI).

图 1.细胞核图像分割的性能比较。语义分割通过Dice系数测量，实例分割（标记为inst.）通过聚合杰卡德指数（Aggregated Jaccard Index, AJI）测量。

Fig. 2. The overview of the proposed UN-SAM for nuclei image segmentation, consisting of DT-Encoder, SPGen and DQ-Decoder. For ease of understanding, we elaborate on the case of UN-SAM with four nuclei image domains. Our UN-SAM can achieve superior generalization performance on these domains without the need for manual annotations.

图 2 展示了提出的UN-SAM（用于细胞核图像分割）的概览，包括DT-Encoder、SPGen和DQ-Decoder。为了便于理解，我们详细介绍了UN-SAM在四个细胞核图像领域的情况。我们的UN-SAM能够在这些领域上实现卓越的泛化性能，而无需手动注释。

Fig. 3. The illustration of the transformer layer with the domain bypass in the DT-Encoder. In the domain bypass, the domain-common and domain-specific embeddings process image embeddings in sequence.

图 3. 展示了DT-Encoder中具有领域旁路的变换器层的插图。在领域旁路中，领域通用和领域特定的嵌入按顺序处理图像嵌入。

Fig. 4. Visualization of generalized nuclei instance segmentation. + indicates medical SAMs using point prompts, and ∗ indicates that classical methods are trained and tested separately for each dataset.

Our UN-SAM segments more nuclei with accurate boundaries while having fewer false positives.

图 4.展示了泛化的细胞核实例分割的可视化。

+ 表示使用点提示的医学SAMs，而 ∗ 表示传统方法对每个数据集分别进行训练和测试。我们的UN-SAM在减少假阳性的同时，能够分割出更多具有准确边界的细胞核。

Fig. 5. Hyper-parameter analysis of confidence threshold on nuclei semantic segmentation of the DSB dataset.

图 5. DSB数据集上细胞核语义分割的置信度阈值的超参数分析。

Fig. 6. Visualization of generalized nuclei semantic segmentation. + indicates medical SAMs using point prompts, and ∗ indicates that classical methods are trained and tested separately for each dataset. Our UN-SAM exhibits the best results, segmenting more nuclei with accurate boundaries while having fewer false positives. The SPGen produces the self-prompt to provide high-quality segmentation hints for our UN-SAM.

图 6 展示了泛化的细胞核语义分割的可视化。+ 表示使用点提示的医学SAMs，而 ∗ 表示传统方法对每个数据集分别进行训练和测试。我们的UN-SAM展示了最佳结果，在减少假阳性的同时，能够分割出更多具有准确边界的细胞核。SPGen生成自我提示，为我们的UN-SAM提供高质量的分割提示。

TABLE

表