  Labeling videos at scale is impractical. Consequently, self-supervised visual
representation learning is key for efficient video analysis. Recent success in
learning image representations suggests contrastive learning is a promising
framework to tackle this challenge. However, when applied to real-world videos,
contrastive learning may unknowingly lead to the separation of instances that
contain semantically similar events. In our work, we introduce a cooperative
variant of contrastive learning to utilize complementary information across
views and address this issue. We use data-driven sampling to leverage implicit
relationships between multiple input video views, whether observed (e.g. RGB)
or inferred (e.g. flow, segmentation masks, poses). We are one of the firsts to
explore exploiting inter-instance relationships to drive learning. We
experimentally evaluate our representations on the downstream task of action
recognition. Our method achieves competitive performance on standard benchmarks
(UCF101, HMDB51, Kinetics400). Furthermore, qualitative experiments illustrate
that our models can capture higher-order class relationships.
