  In this paper, we aim to reduce the computational cost of spatio-temporal
deep neural networks, making them run as fast as their 2D counterparts while
preserving state-of-the-art accuracy on video recognition benchmarks. To this
end, we present the novel Multi-Fiber architecture that slices a complex neural
network into an ensemble of lightweight networks or fibers that run through the
network. To facilitate information flow between fibers we further incorporate
multiplexer modules and end up with an architecture that reduces the
computational cost of 3D networks by an order of magnitude, while increasing
recognition performance at the same time. Extensive experimental results show
that our multi-fiber architecture significantly boosts the efficiency of
existing convolution networks for both image and video recognition tasks,
achieving state-of-the-art performance on UCF-101, HMDB-51 and Kinetics
datasets. Our proposed model requires over 9x and 13x less computations than
the I3D and R(2+1)D models, respectively, yet providing higher accuracy.
