  U-Nets have achieved tremendous success in medical image segmentation.
Nevertheless, it may suffer limitations in global (long-range) contextual
interactions and edge-detail preservation. In contrast, Transformer has an
excellent ability to capture long-range dependencies by leveraging the
self-attention mechanism into the encoder. Although Transformer was born to
model the long-range dependency on the extracted feature maps, it still suffers
from extreme computational and spatial complexities in processing
high-resolution 3D feature maps. This motivates us to design the efficiently
Transformer-based UNet model and study the feasibility of Transformer-based
network architectures for medical image segmentation tasks. To this end, we
propose to self-distill a Transformer-based UNet for medical image
segmentation, which simultaneously learns global semantic information and local
spatial-detailed features. Meanwhile, a local multi-scale fusion block is first
proposed to refine fine-grained details from the skipped connections in the
encoder by the main CNN stem through self-distillation, only computed during
training and removed at inference with minimal overhead. Extensive experiments
on BraTS 2019 and CHAOS datasets show that our MISSU achieves the best
performance over previous state-of-the-art methods. Code and models are
available at \url{https://github.com/wangn123/MISSU.git}
