  Learning discriminative spatiotemporal representation is the key problem of
video understanding. Recently, Vision Transformers (ViTs) have shown their
power in learning long-term video dependency with self-attention.
Unfortunately, they exhibit limitations in tackling local video redundancy, due
to the blind global comparison among tokens. UniFormer has successfully
alleviated this issue, by unifying convolution and self-attention as a relation
aggregator in the transformer format. However, this model has to require a
tiresome and complicated image-pretraining phrase, before being finetuned on
videos. This blocks its wide usage in practice. On the contrary, open-sourced
ViTs are readily available and well-pretrained with rich image supervision.
Based on these observations, we propose a generic paradigm to build a powerful
family of video networks, by arming the pretrained ViTs with efficient
UniFormer designs. We call this family UniFormerV2, since it inherits the
concise style of the UniFormer block. But it contains brand-new local and
global relation aggregators, which allow for preferable accuracy-computation
balance by seamlessly integrating advantages from both ViTs and UniFormer.
Without any bells and whistles, our UniFormerV2 gets the state-of-the-art
recognition performance on 8 popular video benchmarks, including scene-related
Kinetics-400/600/700 and Moments in Time, temporal-related Something-Something
V1/V2, untrimmed ActivityNet and HACS. In particular, it is the first model to
achieve 90% top-1 accuracy on Kinetics-400, to our best knowledge. Code will be
available at https://github.com/OpenGVLab/UniFormerV2.
