  Convolutional Neural Networks are the de facto models for image recognition.
However 3D CNNs, the straight forward extension of 2D CNNs for video
recognition, have not achieved the same success on standard action recognition
benchmarks. One of the main reasons for this reduced performance of 3D CNNs is
the increased computational complexity requiring large scale annotated datasets
to train them in scale. 3D kernel factorization approaches have been proposed
to reduce the complexity of 3D CNNs. Existing kernel factorization approaches
follow hand-designed and hard-wired techniques. In this paper we propose
Gate-Shift-Fuse (GSF), a novel spatio-temporal feature extraction module which
controls interactions in spatio-temporal decomposition and learns to adaptively
route features through time and combine them in a data dependent manner. GSF
leverages grouped spatial gating to decompose input tensor and channel
weighting to fuse the decomposed tensors. GSF can be inserted into existing 2D
CNNs to convert them into an efficient and high performing spatio-temporal
feature extractor, with negligible parameter and compute overhead. We perform
an extensive analysis of GSF using two popular 2D CNN families and achieve
state-of-the-art or competitive performance on five standard action recognition
benchmarks.
