  With the rapid development of digital multimedia, video understanding has
become an important field. For action recognition, temporal dimension plays an
important role, and this is quite different from image recognition. In order to
learn powerful feature of videos, we propose a Collaborative Temporal Modeling
(CTM) block (Figure 1) to learn temporal information for action recognition.
Besides a parameter-free identity shortcut, as a separate temporal modeling
block, CTM includes two collaborative paths: a spatial-aware temporal modeling
path, which we propose the Temporal-Channel Convolution Module (TCCM) with
unshared parameters for each spatial position (H*W) to build, and a
spatial-unaware temporal modeling path. CTM blocks can seamlessly be inserted
into many popular networks to generate CTM Networks and bring the capability of
learning temporal information to 2D CNN backbone networks, which only capture
spatial information. Experiments on several popular action recognition datasets
demonstrate that CTM blocks bring the performance improvements on 2D CNN
baselines, and our method achieves the competitive results against the
state-of-the-art methods. Code will be made publicly available.
