  Attention mechanisms have significantly boosted the performance of video
classification neural networks thanks to the utilization of perspective
contexts. However, the current research on video attention generally focuses on
adopting a specific aspect of contexts (e.g., channel, spatial/temporal, or
global context) to refine the features and neglects their underlying
correlation when computing attentions. This leads to incomplete context
utilization and hence bears the weakness of limited performance improvement. To
tackle the problem, this paper proposes an efficient attention-in-attention
(AIA) method for element-wise feature refinement, which investigates the
feasibility of inserting the channel context into the spatio-temporal attention
learning module, referred to as CinST, and also its reverse variant, referred
to as STinC. Specifically, we instantiate the video feature contexts as
dynamics aggregated along a specific axis with global average and max pooling
operations. The workflow of an AIA module is that the first attention block
uses one kind of context information to guide the gating weights calculation of
the second attention that targets at the other context. Moreover, all the
computational operations in attention units act on the pooled dimension, which
results in quite few computational cost increase ($<$0.02\%). To verify our
method, we densely integrate it into two classical video network backbones and
conduct extensive experiments on several standard video classification
benchmarks. The source code of our AIA is available at
\url{https://github.com/haoyanbin918/Attention-in-Attention}.
