  Vision transformers have recently emerged as an effective alternative to
convolutional networks for action recognition. However, vision transformers
still struggle with geometric variations prevalent in video data. This paper
proposes a novel approach, GeoDeformer, designed to capture the variations
inherent in action video by integrating geometric comprehension directly into
the ViT architecture. Specifically, at the core of GeoDeformer is the Geometric
Deformation Predictor, a module designed to identify and quantify potential
spatial and temporal geometric deformations within the given video. Spatial
deformations adjust the geometry within individual frames, while temporal
deformations capture the cross-frame geometric dynamics, reflecting motion and
temporal progression. To demonstrate the effectiveness of our approach, we
incorporate it into the established MViTv2 framework, replacing the standard
self-attention blocks with GeoDeformer blocks. Our experiments at UCF101,
HMDB51, and Mini-K200 achieve significant increases in both Top-1 and Top-5
accuracy, establishing new state-of-the-art results with only a marginal
increase in computational cost. Additionally, visualizations affirm that
GeoDeformer effectively manifests explicit geometric deformations and minimizes
geometric variations. Codes and checkpoints will be released.
