  Deep neural networks require collecting and annotating large amounts of data
to train successfully. In order to alleviate the annotation bottleneck, we
propose a novel self-supervised representation learning approach for
spatiotemporal features extracted from videos. We introduce Skip-Clip, a method
that utilizes temporal coherence in videos, by training a deep model for future
clip order ranking conditioned on a context clip as a surrogate objective for
video future prediction. We show that features learned using our method are
generalizable and transfer strongly to downstream tasks. For action recognition
on the UCF101 dataset, we obtain 51.8% improvement over random initialization
and outperform models initialized using inflated ImageNet parameters. Skip-Clip
also achieves results competitive with state-of-the-art self-supervision
methods.
