  Local Transformer-based classification models have recently achieved
promising results with relatively low computational costs. However, the effect
of aggregating spatial global information of local Transformer-based
architecture is not clear. This work investigates the outcome of applying a
global attention-based module named multi-resolution overlapped attention (MOA)
in the local window-based transformer after each stage. The proposed MOA
employs slightly larger and overlapped patches in the key to enable
neighborhood pixel information transmission, which leads to significant
performance gain. In addition, we thoroughly investigate the effect of the
dimension of essential architecture components through extensive experiments
and discover an optimum architecture design. Extensive experimental results
CIFAR-10, CIFAR-100, and ImageNet-1K datasets demonstrate that the proposed
approach outperforms previous vision Transformers with a comparatively fewer
number of parameters.
