  We propose SCVRL, a novel contrastive-based framework for self-supervised
learning for videos. Differently from previous contrast learning based methods
that mostly focus on learning visual semantics (e.g., CVRL), SCVRL is capable
of learning both semantic and motion patterns. For that, we reformulate the
popular shuffling pretext task within a modern contrastive learning paradigm.
We show that our transformer-based network has a natural capacity to learn
motion in self-supervised settings and achieves strong performance,
outperforming CVRL on four benchmarks.
