  Robust video scene classification models should capture the spatial
(pixel-wise) and temporal (frame-wise) characteristics of a video effectively.
Transformer models with self-attention which are designed to get contextualized
representations for individual tokens given a sequence of tokens, are becoming
increasingly popular in many computer vision tasks. However, the use of
Transformer based models for video understanding is still relatively
unexplored. Moreover, these models fail to exploit the strong temporal
relationships between the neighboring video frames to get potent frame-level
representations. In this paper, we propose a novel self-attention block that
leverages both local and global temporal relationships between the video frames
to obtain better contextualized representations for the individual frames. This
enables the model to understand the video at various granularities. We
illustrate the performance of our models on the large scale YoutTube-8M data
set on the task of video categorization and further analyze the results to
showcase improvement.
