  Recently, there emerges a series of vision Transformers, which show superior
performance with a more compact model size than conventional convolutional
neural networks, thanks to the strong ability of Transformers to model
long-range dependencies. However, the advantages of vision Transformers also
come with a price: Self-attention, the core part of Transformer, has a
quadratic complexity to the input sequence length. This leads to a dramatic
increase of computation and memory cost with the increase of sequence length,
thus introducing difficulties when applying Transformers to the vision tasks
that require dense predictions based on high-resolution feature maps. In this
paper, we propose a new vision Transformer, named Glance-and-Gaze Transformer
(GG-Transformer), to address the aforementioned issues. It is motivated by the
Glance and Gaze behavior of human beings when recognizing objects in natural
scenes, with the ability to efficiently model both long-range dependencies and
local context. In GG-Transformer, the Glance and Gaze behavior is realized by
two parallel branches: The Glance branch is achieved by performing
self-attention on the adaptively-dilated partitions of the input, which leads
to a linear complexity while still enjoying a global receptive field; The Gaze
branch is implemented by a simple depth-wise convolutional layer, which
compensates local image context to the features obtained by the Glance
mechanism. We empirically demonstrate our method achieves consistently superior
performance over previous state-of-the-art Transformers on various vision tasks
and benchmarks. The codes and models will be made available at
https://github.com/yucornetto/GG-Transformer.
