  While a large number of recent works on semantic segmentation focus on
designing and incorporating a transformer-based encoder, much less attention
and vigor have been devoted to transformer-based decoders. For such a task
whose hallmark quest is pixel-accurate prediction, we argue that the decoder
stage is just as crucial as that of the encoder in achieving superior
segmentation performance, by disentangling and refining the high-level cues and
working out object boundaries with pixel-level precision. In this paper, we
propose a novel transformer-based decoder called UperFormer, which is
plug-and-play for hierarchical encoders and attains high quality segmentation
results regardless of encoder architecture. UperFormer is equipped with
carefully designed multi-head skip attention units and novel upsampling
operations. Multi-head skip attention is able to fuse multi-scale features from
backbones with those in decoders. The upsampling operation, which incorporates
feature from encoder, can be more friendly for object localization. It brings a
0.4% to 3.2% increase compared with traditional upsampling methods. By
combining UperFormer with Swin Transformer (Swin-T), a fully transformer-based
symmetric network is formed for semantic segmentation tasks. Extensive
experiments show that our proposed approach is highly effective and
computationally efficient. On Cityscapes dataset, we achieve state-of-the-art
performance. On the more challenging ADE20K dataset, our best model yields a
single-scale mIoU of 50.18, and a multi-scale mIoU of 51.8, which is on-par
with the current state-of-art model, while we drastically cut the number of
FLOPs by 53.5%. Our source code and models are publicly available at:
https://github.com/shiwt03/UperFormer
