  Instance-level contrastive learning techniques, which rely on data
augmentation and a contrastive loss function, have found great success in the
domain of visual representation learning. They are not suitable for exploiting
the rich dynamical structure of video however, as operations are done on many
augmented instances. In this paper we propose "Video Cross-Stream Prototypical
Contrasting", a novel method which predicts consistent prototype assignments
from both RGB and optical flow views, operating on sets of samples.
Specifically, we alternate the optimization process; while optimizing one of
the streams, all views are mapped to one set of stream prototype vectors. Each
of the assignments is predicted with all views except the one matching the
prediction, pushing representations closer to their assigned prototypes. As a
result, more efficient video embeddings with ingrained motion information are
learned, without the explicit need for optical flow computation during
inference. We obtain state-of-the-art results on nearest-neighbour video
retrieval and action recognition, outperforming previous best by +3.2% on
UCF101 using the S3D backbone (90.5% Top-1 acc), and by +7.2% on UCF101 and
+15.1% on HMDB51 using the R(2+1)D backbone.
