  3D CNN shows its strong ability in learning spatiotemporal representation in
recent video recognition tasks. However, inflating 2D convolution to 3D
inevitably introduces additional computational costs, making it cumbersome in
practical deployment. We consider whether there is a way to equip the
conventional 2D convolution with temporal vision no requiring expanding its
kernel. To this end, we propose the video shuffle, a parameter-free plug-in
component that efficiently reallocates the inputs of 2D convolution so that its
receptive field can be extended to the temporal dimension. In practical, video
shuffle firstly divides each frame feature into multiple groups and then
aggregate the grouped features via temporal shuffle operation. This allows the
following 2D convolution aggregate the global spatiotemporal features. The
proposed video shuffle can be flexibly inserted into popular 2D CNNs, forming
the Video Shuffle Networks (VSN). With a simple yet efficient implementation,
VSN performs surprisingly well on temporal modeling benchmarks. In experiments,
VSN not only gains non-trivial improvements on Kinetics and Moments in Time,
but also achieves state-of-the-art performance on Something-Something-V1,
Something-Something-V2 datasets.
