  Self-supervised learning has made unsupervised pretraining relevant again for
difficult computer vision tasks. The most effective self-supervised methods
involve prediction tasks based on features extracted from diverse views of the
data. DeepInfoMax (DIM) is a self-supervised method which leverages the
internal structure of deep networks to construct such views, forming prediction
tasks between local features which depend on small patches in an image and
global features which depend on the whole image. In this paper, we extend DIM
to the video domain by leveraging similar structure in spatio-temporal
networks, producing a method we call Video Deep InfoMax(VDIM). We find that
drawing views from both natural-rate sequences and temporally-downsampled
sequences yields results on Kinetics-pretrained action recognition tasks which
match or outperform prior state-of-the-art methods that use more costly
large-time-scale transformer models. We also examine the effects of data
augmentation and fine-tuning methods, accomplishingSoTA by a large margin when
training only on the UCF-101 dataset.
