  Motion, as the uniqueness of a video, has been critical to the development of
video understanding models. Modern deep learning models leverage motion by
either executing spatio-temporal 3D convolutions, factorizing 3D convolutions
into spatial and temporal convolutions separately, or computing self-attention
along temporal dimension. The implicit assumption behind such successes is that
the feature maps across consecutive frames can be nicely aggregated.
Nevertheless, the assumption may not always hold especially for the regions
with large deformation. In this paper, we present a new recipe of inter-frame
attention block, namely Stand-alone Inter-Frame Attention (SIFA), that novelly
delves into the deformation across frames to estimate local self-attention on
each spatial location. Technically, SIFA remoulds the deformable design via
re-scaling the offset predictions by the difference between two frames. Taking
each spatial location in the current frame as the query, the locally deformable
neighbors in the next frame are regarded as the keys/values. Then, SIFA
measures the similarity between query and keys as stand-alone attention to
weighted average the values for temporal aggregation. We further plug SIFA
block into ConvNets and Vision Transformer, respectively, to devise SIFA-Net
and SIFA-Transformer. Extensive experiments conducted on four video datasets
demonstrate the superiority of SIFA-Net and SIFA-Transformer as stronger
backbones. More remarkably, SIFA-Transformer achieves an accuracy of 83.1% on
Kinetics-400 dataset. Source code is available at
\url{https://github.com/FuchenUSTC/SIFA}.
