  Videos are a rich source for self-supervised learning (SSL) of visual
representations due to the presence of natural temporal transformations of
objects. However, current methods typically randomly sample video clips for
learning, which results in an imperfect supervisory signal. In this work, we
propose PreViTS, an SSL framework that utilizes an unsupervised tracking signal
for selecting clips containing the same object, which helps better utilize
temporal transformations of objects. PreViTS further uses the tracking signal
to spatially constrain the frame regions to learn from and trains the model to
locate meaningful objects by providing supervision on Grad-CAM attention maps.
To evaluate our approach, we train a momentum contrastive (MoCo) encoder on
VGG-Sound and Kinetics-400 datasets with PreViTS. Training with PreViTS
outperforms representations learnt by contrastive strategy alone on video
downstream tasks, obtaining state-of-the-art performance on action
classification. PreViTS helps learn feature representations that are more
robust to changes in background and context, as seen by experiments on datasets
with background changes. Learning from large-scale videos with PreViTS could
lead to more accurate and robust visual feature representations.
