  The goal of this paper is to self-train a 3D convolutional neural network on
an unlabeled video collection for deployment on small-scale video collections.
As smaller video datasets benefit more from motion than appearance, we strive
to train our network using optical flow, but avoid its computation during
inference. We propose the first motion-augmented self-training regime, we call
MotionFit. We start with supervised training of a motion model on a small, and
labeled, video collection. With the motion model we generate pseudo-labels for
a large unlabeled video collection, which enables us to transfer knowledge by
learning to predict these pseudo-labels with an appearance model. Moreover, we
introduce a multi-clip loss as a simple yet efficient way to improve the
quality of the pseudo-labeling, even without additional auxiliary tasks. We
also take into consideration the temporal granularity of videos during
self-training of the appearance model, which was missed in previous works. As a
result we obtain a strong motion-augmented representation model suited for
video downstream tasks like action recognition and clip retrieval. On
small-scale video datasets, MotionFit outperforms alternatives for knowledge
transfer by 5%-8%, video-only self-supervision by 1%-7% and semi-supervised
learning by 9%-18% using the same amount of class labels.
