  Contrastive learning has shown great potential in video representation
learning. However, existing approaches fail to sufficiently exploit short-term
motion dynamics, which are crucial to various down-stream video understanding
tasks. In this paper, we propose Motion Sensitive Contrastive Learning (MSCL)
that injects the motion information captured by optical flows into RGB frames
to strengthen feature learning. To achieve this, in addition to clip-level
global contrastive learning, we develop Local Motion Contrastive Learning
(LMCL) with frame-level contrastive objectives across the two modalities.
Moreover, we introduce Flow Rotation Augmentation (FRA) to generate extra
motion-shuffled negative samples and Motion Differential Sampling (MDS) to
accurately screen training samples. Extensive experiments on standard
benchmarks validate the effectiveness of the proposed method. With the
commonly-used 3D ResNet-18 as the backbone, we achieve the top-1 accuracies of
91.5\% on UCF101 and 50.3\% on Something-Something v2 for video classification,
and a 65.6\% Top-1 Recall on UCF101 for video retrieval, notably improving the
state-of-the-art.
