  Self-attention learns pairwise interactions to model long-range dependencies,
yielding great improvements for video action recognition. In this paper, we
seek a deeper understanding of self-attention for temporal modeling in videos.
We first demonstrate that the entangled modeling of spatio-temporal information
by flattening all pixels is sub-optimal, failing to capture temporal
relationships among frames explicitly. To this end, we introduce Global
Temporal Attention (GTA), which performs global temporal attention on top of
spatial attention in a decoupled manner. We apply GTA on both pixels and
semantically similar regions to capture temporal relationships at different
levels of spatial granularity. Unlike conventional self-attention that computes
an instance-specific attention matrix, GTA directly learns a global attention
matrix that is intended to encode temporal structures that generalize across
different samples. We further augment GTA with a cross-channel multi-head
fashion to exploit channel interactions for better temporal modeling. Extensive
experiments on 2D and 3D networks demonstrate that our approach consistently
enhances temporal modeling and provides state-of-the-art performance on three
video action recognition datasets.
