  In recent years, vision Transformers and MLPs have demonstrated remarkable
performance in image understanding tasks. However, their inherently dense
computational operators, such as self-attention and token-mixing layers, pose
significant challenges when applied to spatio-temporal video data. To address
this gap, we propose PosMLP-Video, a lightweight yet powerful MLP-like backbone
for video recognition. Instead of dense operators, we use efficient relative
positional encoding (RPE) to build pairwise token relations, leveraging
small-sized parameterized relative position biases to obtain each relation
score. Specifically, to enable spatio-temporal modeling, we extend the image
PosMLP's positional gating unit to temporal, spatial, and spatio-temporal
variants, namely PoTGU, PoSGU, and PoSTGU, respectively. These gating units can
be feasibly combined into three types of spatio-temporal factorized positional
MLP blocks, which not only decrease model complexity but also maintain good
performance. Additionally, we enrich relative positional relationships by using
channel grouping. Experimental results on three video-related tasks demonstrate
that PosMLP-Video achieves competitive speed-accuracy trade-offs compared to
the previous state-of-the-art models. In particular, PosMLP-Video pre-trained
on ImageNet1K achieves 59.0%/70.3% top-1 accuracy on Something-Something V1/V2
and 82.1% top-1 accuracy on Kinetics-400 while requiring much fewer parameters
and FLOPs than other models. The code is released at
https://github.com/zhouds1918/PosMLP_Video.
