  We introduce Video Transformer (VidTr) with separable-attention for video
classification. Comparing with commonly used 3D networks, VidTr is able to
aggregate spatio-temporal information via stacked attentions and provide better
performance with higher efficiency. We first introduce the vanilla video
transformer and show that transformer module is able to perform spatio-temporal
modeling from raw pixels, but with heavy memory usage. We then present VidTr
which reduces the memory cost by 3.3$\times$ while keeping the same
performance. To further optimize the model, we propose the standard deviation
based topK pooling for attention ($pool_{topK\_std}$), which reduces the
computation by dropping non-informative features along temporal dimension.
VidTr achieves state-of-the-art performance on five commonly used datasets with
lower computational requirement, showing both the efficiency and effectiveness
of our design. Finally, error analysis and visualization show that VidTr is
especially good at predicting actions that require long-term temporal
reasoning.
