  It is not trivial to optimally learn a 3D Convolutional Neural Networks (3D
ConvNets) due to high complexity and various options of the training scheme.
The most common hand-tuning process starts from learning 3D ConvNets using
short video clips and then is followed by learning long-term temporal
dependency using lengthy clips, while gradually decaying the learning rate from
high to low as training progresses. The fact that such process comes along with
several heuristic settings motivates the study to seek an optimal "path" to
automate the entire training. In this paper, we decompose the path into a
series of training "states" and specify the hyper-parameters, e.g., learning
rate and the length of input clips, in each state. The estimation of the knee
point on the performance-epoch curve triggers the transition from one state to
another. We perform dynamic programming over all the candidate states to plan
the optimal permutation of states, i.e., optimization path. Furthermore, we
devise a new 3D ConvNets with a unique design of dual-head classifier to
improve spatial and temporal discrimination. Extensive experiments on seven
public video recognition benchmarks demonstrate the advantages of our proposal.
With the optimization planning, our 3D ConvNets achieves superior results when
comparing to the state-of-the-art recognition methods. More remarkably, we
obtain the top-1 accuracy of 80.5% and 82.7% on Kinetics-400 and Kinetics-600
datasets, respectively. Source code is available at
https://github.com/ZhaofanQiu/Optimization-Planning-for-3D-ConvNets.
