  This paper is on video recognition using Transformers. Very recent attempts
in this area have demonstrated promising results in terms of recognition
accuracy, yet they have been also shown to induce, in many cases, significant
computational overheads due to the additional modelling of the temporal
information. In this work, we propose a Video Transformer model the complexity
of which scales linearly with the number of frames in the video sequence and
hence induces no overhead compared to an image-based Transformer model. To
achieve this, our model makes two approximations to the full space-time
attention used in Video Transformers: (a) It restricts time attention to a
local temporal window and capitalizes on the Transformer's depth to obtain full
temporal coverage of the video sequence. (b) It uses efficient space-time
mixing to attend jointly spatial and temporal locations without inducing any
additional cost on top of a spatial-only attention model. We also show how to
integrate 2 very lightweight mechanisms for global temporal-only attention
which provide additional accuracy improvements at minimal computational cost.
We demonstrate that our model produces very high recognition accuracy on the
most popular video recognition datasets while at the same time being
significantly more efficient than other Video Transformer models. Code will be
made available.
