  Transformers have become one of the dominant architectures in deep learning,
particularly as a powerful alternative to convolutional neural networks (CNNs)
in computer vision. However, Transformer training and inference in previous
works can be prohibitively expensive due to the quadratic complexity of
self-attention over a long sequence of representations, especially for
high-resolution dense prediction tasks. To this end, we present a novel Less
attention vIsion Transformer (LIT), building upon the fact that the early
self-attention layers in Transformers still focus on local patterns and bring
minor benefits in recent hierarchical vision Transformers. Specifically, we
propose a hierarchical Transformer where we use pure multi-layer perceptrons
(MLPs) to encode rich local patterns in the early stages while applying
self-attention modules to capture longer dependencies in deeper layers.
Moreover, we further propose a learned deformable token merging module to
adaptively fuse informative patches in a non-uniform manner. The proposed LIT
achieves promising performance on image recognition tasks, including image
classification, object detection and instance segmentation, serving as a strong
backbone for many vision tasks. Code is available at:
https://github.com/zhuang-group/LIT
