  Semantic segmentation has witnessed remarkable advancements with the
adaptation of the Transformer architecture. Parallel to the strides made by the
Transformer, CNN-based U-Net has seen significant progress, especially in
high-resolution medical imaging and remote sensing. This dual success inspired
us to merge the strengths of both, leading to the inception of a U-Net-based
vision transformer decoder tailored for efficient contextual encoding. Here, we
propose a novel transformer decoder, U-MixFormer, built upon the U-Net
structure, designed for efficient semantic segmentation. Our approach
distinguishes itself from the previous transformer methods by leveraging
lateral connections between the encoder and decoder stages as feature queries
for the attention modules, apart from the traditional reliance on skip
connections. Moreover, we innovatively mix hierarchical feature maps from
various encoder and decoder stages to form a unified representation for keys
and values, giving rise to our unique mix-attention module. Our approach
demonstrates state-of-the-art performance across various configurations.
Extensive experiments show that U-MixFormer outperforms SegFormer, FeedFormer,
and SegNeXt by a large margin. For example, U-MixFormer-B0 surpasses
SegFormer-B0 and FeedFormer-B0 with 3.8% and 2.0% higher mIoU and 27.3% and
21.8% less computation and outperforms SegNext with 3.3% higher mIoU with
MSCAN-T encoder on ADE20K. Code available at
https://github.com/julian-klitzing/u-mixformer.
