  Transformers have shown great potential in various computer vision tasks
owing to their strong capability in modeling long-range dependency using the
self-attention mechanism. Nevertheless, vision transformers treat an image as
1D sequence of visual tokens, lacking an intrinsic inductive bias (IB) in
modeling local visual structures and dealing with scale variance.
Alternatively, they require large-scale training data and longer training
schedules to learn the IB implicitly. In this paper, we propose a novel Vision
Transformer Advanced by Exploring intrinsic IB from convolutions, ie, ViTAE.
Technically, ViTAE has several spatial pyramid reduction modules to downsample
and embed the input image into tokens with rich multi-scale context by using
multiple convolutions with different dilation rates. In this way, it acquires
an intrinsic scale invariance IB and is able to learn robust feature
representation for objects at various scales. Moreover, in each transformer
layer, ViTAE has a convolution block in parallel to the multi-head
self-attention module, whose features are fused and fed into the feed-forward
network. Consequently, it has the intrinsic locality IB and is able to learn
local features and global dependencies collaboratively. Experiments on ImageNet
as well as downstream tasks prove the superiority of ViTAE over the baseline
transformer and concurrent works. Source code and pretrained models will be
available at GitHub.
