  For video recognition task, a global representation summarizing the whole
contents of the video snippets plays an important role for the final
performance. However, existing video architectures usually generate it by using
a simple, global average pooling (GAP) method, which has limited ability to
capture complex dynamics of videos. For image recognition task, there exist
evidences showing that covariance pooling has stronger representation ability
than GAP. Unfortunately, such plain covariance pooling used in image
recognition is an orderless representative, which cannot model spatio-temporal
structure inherent in videos. Therefore, this paper proposes a
Temporal-attentive Covariance Pooling(TCP), inserted at the end of deep
architectures, to produce powerful video representations. Specifically, our TCP
first develops a temporal attention module to adaptively calibrate
spatio-temporal features for the succeeding covariance pooling, approximatively
producing attentive covariance representations. Then, a temporal covariance
pooling performs temporal pooling of the attentive covariance representations
to characterize both intra-frame correlations and inter-frame
cross-correlations of the calibrated features. As such, the proposed TCP can
capture complex temporal dynamics. Finally, a fast matrix power normalization
is introduced to exploit geometry of covariance representations. Note that our
TCP is model-agnostic and can be flexibly integrated into any video
architectures, resulting in TCPNet for effective video recognition. The
extensive experiments on six benchmarks (e.g., Kinetics, Something-Something V1
and Charades) using various video architectures show our TCPNet is clearly
superior to its counterparts, while having strong generalization ability. The
source code is publicly available.
