  Recent advancements in vision backbones have significantly improved their
performance by simultaneously modeling images' local and global contexts.
However, the bidirectional interaction between these two contexts has not been
well explored and exploited, which is important in the human visual system.
This paper proposes a Fully Adaptive Self-Attention (FASA) mechanism for vision
transformer to model the local and global information as well as the
bidirectional interaction between them in context-aware ways. Specifically,
FASA employs self-modulated convolutions to adaptively extract local
representation while utilizing self-attention in down-sampled space to extract
global representation. Subsequently, it conducts a bidirectional adaptation
process between local and global representation to model their interaction. In
addition, we introduce a fine-grained downsampling strategy to enhance the
down-sampled self-attention mechanism for finer-grained global perception
capability. Based on FASA, we develop a family of lightweight vision backbones,
Fully Adaptive Transformer (FAT) family. Extensive experiments on multiple
vision tasks demonstrate that FAT achieves impressive performance. Notably, FAT
accomplishes a 77.6% accuracy on ImageNet-1K using only 4.5M parameters and
0.7G FLOPs, which surpasses the most advanced ConvNets and Transformers with
similar model size and computational costs. Moreover, our model exhibits faster
speed on modern GPU compared to other models. Code will be available at
https://github.com/qhfan/FAT.
