  Self-attention mechanism has been a key factor in the recent progress of
Vision Transformer (ViT), which enables adaptive feature extraction from global
contexts. However, existing self-attention methods either adopt sparse global
attention or window attention to reduce the computation complexity, which may
compromise the local feature learning or subject to some handcrafted designs.
In contrast, local attention, which restricts the receptive field of each query
to its own neighboring pixels, enjoys the benefits of both convolution and
self-attention, namely local inductive bias and dynamic feature selection.
Nevertheless, current local attention modules either use inefficient Im2Col
function or rely on specific CUDA kernels that are hard to generalize to
devices without CUDA support. In this paper, we propose a novel local attention
module, Slide Attention, which leverages common convolution operations to
achieve high efficiency, flexibility and generalizability. Specifically, we
first re-interpret the column-based Im2Col function from a new row-based
perspective and use Depthwise Convolution as an efficient substitution. On this
basis, we propose a deformed shifting module based on the re-parameterization
technique, which further relaxes the fixed key/value positions to deformed
features in the local region. In this way, our module realizes the local
attention paradigm in both efficient and flexible manner. Extensive experiments
show that our slide attention module is applicable to a variety of advanced
Vision Transformer models and compatible with various hardware devices, and
achieves consistently improved performances on comprehensive benchmarks. Code
is available at https://github.com/LeapLabTHU/Slide-Transformer.
