  Contrastive learning has revolutionized self-supervised image representation
learning field, and recently been adapted to video domain. One of the greatest
advantages of contrastive learning is that it allows us to flexibly define
powerful loss objectives as long as we can find a reasonable way to formulate
positive and negative samples to contrast. However, existing approaches rely
heavily on the short-range spatiotemporal salience to form clip-level
contrastive signals, thus limit themselves from using global context. In this
paper, we propose a new video-level contrastive learning method based on
segments to formulate positive pairs. Our formulation is able to capture global
context in a video, thus robust to temporal content change. We also incorporate
a temporal order regularization term to enforce the inherent sequential
structure of videos. Extensive experiments show that our video-level
contrastive learning framework (VCLR) is able to outperform previous
state-of-the-arts on five video datasets for downstream action classification,
action localization and video retrieval. Code is available at
https://github.com/amazon-research/video-contrastive-learning.
