  Recently, Transformers have shown promising performance in various vision
tasks. However, the high costs of global self-attention remain challenging for
Transformers, especially for high-resolution vision tasks. Inspired by one of
the most successful transformers-based models for NLP: Big Bird, we propose a
novel sparse attention mechanism for Vision Transformers (ViT). Specifically,
we separate the heads into three groups, the first group used convolutional
neural network (CNN) to extract local features and provide positional
information for the model, the second group used Random Sampling Windows
(RS-Win) for sparse self-attention calculation, and the third group reduces the
resolution of the keys and values by average pooling for global attention.
Based on these components, ViT maintains the sparsity of self-attention while
maintaining the merits of Big Bird (i.e., the model is a universal approximator
of sequence functions and is Turing complete). Moreover, our results show that
the positional encoding, a crucial component in ViTs, can be safely removed in
our model. Experiments show that Vision Big Bird demonstrates competitive
performance on common vision tasks.
