  Recent work has shown the potential of transformers for computer vision
applications. An image is first partitioned into patches, which are then used
as input tokens for the attention mechanism. Due to the expensive quadratic
cost of the attention mechanism, either a large patch size is used, resulting
in coarse-grained global interactions, or alternatively, attention is applied
only on a local region of the image, at the expense of long-range interactions.
In this work, we propose an approach that allows for both coarse global
interactions and fine-grained local interactions already at early layers of a
vision transformer.
  At the core of our method is the application of local and global attention
layers. In the local attention layer, we apply attention to each patch and its
local shifts, resulting in virtually located local patches, which are not bound
to a single, specific location. These virtually located patches are then used
in a global attention layer. The separation of the attention layer into local
and global counterparts allows for a low computational cost in the number of
patches, while still supporting data-dependent localization already at the
first layer, as opposed to the static positioning in other visual transformers.
Our method is shown to be superior to both convolutional and transformer-based
methods for image classification on CIFAR10, CIFAR100, and ImageNet. Code is
available at: https://github.com/shellysheynin/Locally-SAG-Transformer.
