  The success of Vision Transformer (ViT) has been widely reported on a wide
range of image recognition tasks. ViT can learn global dependencies superior to
CNN, yet CNN's inherent locality can substitute for expensive training
resources. Recently, the outstanding performance of RetNet in the field of
language modeling has garnered attention, surpassing that of the Transformer
with explicit local modeling, shifting researchers' focus towards Transformers
in the CV field. This paper investigates the effectiveness of RetNet from a CNN
perspective and presents a variant of RetNet tailored to the visual domain.
Similar to RetNet we improves ViT's local modeling by applying a weight mask on
the original self-attention matrix. A straightforward way to locally adapt the
self-attention matrix can be realized by an element-wise learnable weight mask
(ELM), for which our preliminary results show promising results. However, the
element-wise simple learnable weight mask not only induces a non-trivial
additional parameter overhead but also increases the optimization complexity.
To this end, this work proposes a novel Gaussian mixture mask (GMM) in which
one mask only has two learnable parameters and it can be conveniently used in
any ViT variants whose attention mechanism allows the use of masks.
Experimental results on multiple small datasets demonstrate that the
effectiveness of our proposed Gaussian mask for boosting ViTs for free (almost
zero additional parameter or computation cost). Our code can be publicly
available at https://github.com/CatworldLee/Gaussian-Mixture-Mask-Attention.
