  Both Convolutional Neural Networks (CNNs) and Transformers have shown great
success in semantic segmentation tasks. Efforts have been made to integrate
CNNs with Transformer models to capture both local and global context
interactions. However, there is still room for enhancement, particularly when
considering constraints on computational resources. In this paper, we introduce
HAFormer, a model that combines the hierarchical features extraction ability of
CNNs with the global dependency modeling capability of Transformers to tackle
lightweight semantic segmentation challenges. Specifically, we design a
Hierarchy-Aware Pixel-Excitation (HAPE) module for adaptive multi-scale local
feature extraction. During the global perception modeling, we devise an
Efficient Transformer (ET) module streamlining the quadratic calculations
associated with traditional Transformers. Moreover, a correlation-weighted
Fusion (cwF) module selectively merges diverse feature representations,
significantly enhancing predictive accuracy. HAFormer achieves high performance
with minimal computational overhead and compact model size, achieving 74.2%
mIoU on Cityscapes and 71.1% mIoU on CamVid test datasets, with frame rates of
105FPS and 118FPS on a single 2080Ti GPU. The source codes are available at
https://github.com/XU-GITHUB-curry/HAFormer.
