  Despite the impressive representation capacity of vision transformer models,
current light-weight vision transformer models still suffer from inconsistent
and incorrect dense predictions at local regions. We suspect that the power of
their self-attention mechanism is limited in shallower and thinner networks. We
propose Lite Vision Transformer (LVT), a novel light-weight transformer network
with two enhanced self-attention mechanisms to improve the model performances
for mobile deployment. For the low-level features, we introduce Convolutional
Self-Attention (CSA). Unlike previous approaches of merging convolution and
self-attention, CSA introduces local self-attention into the convolution within
a kernel of size 3x3 to enrich low-level features in the first stage of LVT.
For the high-level features, we propose Recursive Atrous Self-Attention (RASA),
which utilizes the multi-scale context when calculating the similarity map and
a recursive mechanism to increase the representation capability with marginal
extra parameter cost. The superiority of LVT is demonstrated on ImageNet
recognition, ADE20K semantic segmentation, and COCO panoptic segmentation. The
code is made publicly available.
