  Vision Transformers (ViTs) have emerged with superior performance on computer
vision tasks compared to convolutional neural network (CNN)-based models.
However, ViTs are mainly designed for image classification that generate
single-scale low-resolution representations, which makes dense prediction tasks
such as semantic segmentation challenging for ViTs. Therefore, we propose
HRViT, which enhances ViTs to learn semantically-rich and spatially-precise
multi-scale representations by integrating high-resolution multi-branch
architectures with ViTs. We balance the model performance and efficiency of
HRViT by various branch-block co-optimization techniques. Specifically, we
explore heterogeneous branch designs, reduce the redundancy in linear layers,
and augment the attention block with enhanced expressiveness. Those approaches
enabled HRViT to push the Pareto frontier of performance and efficiency on
semantic segmentation to a new level, as our evaluation results on ADE20K and
Cityscapes show. HRViT achieves 50.20% mIoU on ADE20K and 83.16% mIoU on
Cityscapes, surpassing state-of-the-art MiT and CSWin backbones with an average
of +1.78 mIoU improvement, 28% parameter saving, and 21% FLOPs reduction,
demonstrating the potential of HRViT as a strong vision backbone for semantic
segmentation.
