  The explosive growth in video streaming requires video understanding at high
accuracy and low computation cost. Conventional 2D CNNs are computationally
cheap but cannot capture temporal relationships; 3D CNN-based methods can
achieve good performance but are computationally intensive. In this paper, we
propose a generic and effective Temporal Shift Module (TSM) that enjoys both
high efficiency and high performance. The key idea of TSM is to shift part of
the channels along the temporal dimension, thus facilitate information
exchanged among neighboring frames. It can be inserted into 2D CNNs to achieve
temporal modeling at zero computation and zero parameters. TSM offers several
unique advantages. Firstly, TSM has high performance; it ranks the first on the
Something-Something leaderboard upon submission. Secondly, TSM has high
efficiency; it achieves a high frame rate of 74fps and 29fps for online video
recognition on Jetson Nano and Galaxy Note8. Thirdly, TSM has higher
scalability compared to 3D networks, enabling large-scale Kinetics training on
1,536 GPUs in 15 minutes. Lastly, TSM enables action concepts learning, which
2D networks cannot model; we visualize the category attention map and find that
spatial-temporal action detector emerges during the training of classification
tasks. The code is publicly available at
https://github.com/mit-han-lab/temporal-shift-module.
