  The objective of this paper is visual-only self-supervised video
representation learning. We make the following contributions: (i) we
investigate the benefit of adding semantic-class positives to instance-based
Info Noise Contrastive Estimation (InfoNCE) training, showing that this form of
supervised contrastive learning leads to a clear improvement in performance;
(ii) we propose a novel self-supervised co-training scheme to improve the
popular infoNCE loss, exploiting the complementary information from different
views, RGB streams and optical flow, of the same data source by using one view
to obtain positive class samples for the other; (iii) we thoroughly evaluate
the quality of the learnt representation on two different downstream tasks:
action recognition and video retrieval. In both cases, the proposed approach
demonstrates state-of-the-art or comparable performance with other
self-supervised approaches, whilst being significantly more efficient to train,
i.e. requiring far less training data to achieve similar performance.
