  Deep neural networks have achieved remarkable success for video-based action
recognition. However, most of existing approaches cannot be deployed in
practice due to the high computational cost. To address this challenge, we
propose a new real-time convolutional architecture, called Temporal
Convolutional 3D Network (T-C3D), for action representation. T-C3D learns video
action representations in a hierarchical multi-granularity manner while
obtaining a high process speed. Specifically, we propose a residual 3D
Convolutional Neural Network (CNN) to capture complementary information on the
appearance of a single frame and the motion between consecutive frames. Based
on this CNN, we develop a new temporal encoding method to explore the temporal
dynamics of the whole video. Furthermore, we integrate deep compression
techniques with T-C3D to further accelerate the deployment of models via
reducing the size of the model. By these means, heavy calculations can be
avoided when doing the inference, which enables the method to deal with videos
beyond real-time speed while keeping promising performance. Our method achieves
clear improvements on UCF101 action recognition benchmark against
state-of-the-art real-time methods by 5.4% in terms of accuracy and 2 times
faster in terms of inference speed with a less than 5MB storage model. We
validate our approach by studying its action representation performance on four
different benchmarks over three different tasks. Extensive experiments
demonstrate comparable recognition performance to the state-of-the-art methods.
The source code and the pre-trained models are publicly available at
https://github.com/tc3d.
