  Recently, 3D convolutional networks yield good performance in action
recognition. However, optical flow stream is still needed to ensure better
performance, the cost of which is very high. In this paper, we propose a fast
but effective way to extract motion features from videos utilizing residual
frames as the input data in 3D ConvNets. By replacing traditional stacked RGB
frames with residual ones, 20.5% and 12.5% points improvements over top-1
accuracy can be achieved on the UCF101 and HMDB51 datasets when trained from
scratch. Because residual frames contain little information of object
appearance, we further use a 2D convolutional network to extract appearance
features and combine them with the results from residual frames to form a
two-path solution. In three benchmark datasets, our two-path solution achieved
better or comparable performances than those using additional optical flow
methods, especially outperformed the state-of-the-art models on Mini-kinetics
dataset. Further analysis indicates that better motion features can be
extracted using residual frames with 3D ConvNets, and our residual-frame-input
path is a good supplement for existing RGB-frame-input models.
