  Accurate nuclei segmentation is an essential foundation for various
applications in computational pathology, including cancer diagnosis and
treatment planning. Even slight variations in nuclei representations can
significantly impact these downstream tasks. However, achieving accurate
segmentation remains challenging due to factors like clustered nuclei, high
intra-class variability in size and shape, resemblance to other cells, and
color or contrast variations between nuclei and background. Despite the
extensive utilization of Convolutional Neural Networks (CNNs) in medical image
segmentation, they may have trouble capturing long-range dependencies crucial
for accurate nuclei delineation. Transformers address this limitation but might
miss essential low-level features. To overcome these limitations, we utilized
CNN-Transformer-based techniques for nuclei segmentation in H&E stained
histology images. In this work, we proposed two CNN-Transformer architectures,
Nuclei Hybrid Vision Transformer (NucleiHVT) and Channel Boosted Nuclei Hybrid
Vision Transformer (CB-NucleiHVT), that leverage the strengths of both CNNs and
Transformers to effectively learn nuclei boundaries in multi-organ histology
images. The first architecture, NucleiHVT is inspired by the UNet architecture
and incorporates the dual attention mechanism to capture both multi-level and
multi-scale context effectively. The CB-NucleiHVT network, on the other hand,
utilizes the concept of channel boosting to learn diverse feature spaces,
enhancing the model's ability to distinguish subtle variations in nuclei
characteristics. Detailed evaluation of two medical image segmentation datasets
shows that the proposed architectures outperform existing CNN-based,
Transformer-based, and hybrid methods. The proposed networks demonstrated
effective results both in terms of quantitative metrics, and qualitative visual
assessment.
