  This paper tackles the high computational/space complexity associated with
Multi-Head Self-Attention (MHSA) in vanilla vision transformers. To this end,
we propose Hierarchical MHSA (H-MHSA), a novel approach that computes
self-attention in a hierarchical fashion. Specifically, we first divide the
input image into patches as commonly done, and each patch is viewed as a token.
Then, the proposed H-MHSA learns token relationships within local patches,
serving as local relationship modeling. Then, the small patches are merged into
larger ones, and H-MHSA models the global dependencies for the small number of
the merged tokens. At last, the local and global attentive features are
aggregated to obtain features with powerful representation capacity. Since we
only calculate attention for a limited number of tokens at each step, the
computational load is reduced dramatically. Hence, H-MHSA can efficiently model
global relationships among tokens without sacrificing fine-grained information.
With the H-MHSA module incorporated, we build a family of
Hierarchical-Attention-based Transformer Networks, namely HAT-Net. To
demonstrate the superiority of HAT-Net in scene understanding, we conduct
extensive experiments on fundamental vision tasks, including image
classification, semantic segmentation, object detection, and instance
segmentation. Therefore, HAT-Net provides a new perspective for vision
transformers. Code and pretrained models are available at
https://github.com/yun-liu/HAT-Net.
