  Transformers have made remarkable progress towards modeling long-range
dependencies within the medical image analysis domain. However, current
transformer-based models suffer from several disadvantages: (1) existing
methods fail to capture the important features of the images due to the naive
tokenization scheme; (2) the models suffer from information loss because they
only consider single-scale feature representations; and (3) the segmentation
label maps generated by the models are not accurate enough without considering
rich semantic contexts and anatomical textures. In this work, we present
CASTformer, a novel type of adversarial transformers, for 2D medical image
segmentation. First, we take advantage of the pyramid structure to construct
multi-scale representations and handle multi-scale variations. We then design a
novel class-aware transformer module to better learn the discriminative regions
of objects with semantic structures. Lastly, we utilize an adversarial training
strategy that boosts segmentation accuracy and correspondingly allows a
transformer-based discriminator to capture high-level semantically correlated
contents and low-level anatomical features. Our experiments demonstrate that
CASTformer dramatically outperforms previous state-of-the-art transformer-based
approaches on three benchmarks, obtaining 2.54%-5.88% absolute improvements in
Dice over previous models. Further qualitative experiments provide a more
detailed picture of the model's inner workings, shed light on the challenges in
improved transparency, and demonstrate that transfer learning can greatly
improve performance and reduce the size of medical image datasets in training,
making CASTformer a strong starting point for downstream medical image analysis
tasks.
