  How can we collect and use a video dataset to further improve spatiotemporal
3D Convolutional Neural Networks (3D CNNs)? In order to positively answer this
open question in video recognition, we have conducted an exploration study
using a couple of large-scale video datasets and 3D CNNs. In the early era of
deep neural networks, 2D CNNs have been better than 3D CNNs in the context of
video recognition. Recent studies revealed that 3D CNNs can outperform 2D CNNs
trained on a large-scale video dataset. However, we heavily rely on
architecture exploration instead of dataset consideration. Therefore, in the
present paper, we conduct exploration study in order to improve spatiotemporal
3D CNNs as follows: (i) Recently proposed large-scale video datasets help
improve spatiotemporal 3D CNNs in terms of video classification accuracy. We
reveal that a carefully annotated dataset (e.g., Kinetics-700) effectively
pre-trains a video representation for a video classification task. (ii) We
confirm the relationships between #category/#instance and video classification
accuracy. The results show that #category should initially be fixed, and then
#instance is increased on a video dataset in case of dataset construction.
(iii) In order to practically extend a video dataset, we simply concatenate
publicly available datasets, such as Kinetics-700 and Moments in Time (MiT)
datasets. Compared with Kinetics-700 pre-training, we further enhance
spatiotemporal 3D CNNs with the merged dataset, e.g., +0.9, +3.4, and +1.1 on
UCF-101, HMDB-51, and ActivityNet datasets, respectively, in terms of
fine-tuning. (iv) In terms of recognition architecture, the Kinetics-700 and
merged dataset pre-trained models increase the recognition performance to 200
layers with the Residual Network (ResNet), while the Kinetics-400 pre-trained
model cannot successfully optimize the 200-layer architecture.
