  Pre-training on large-scale video data has become a common recipe for
learning transferable spatiotemporal representations in recent years. Despite
some progress, existing methods are mostly limited to highly curated datasets
(e.g., K400) and exhibit unsatisfactory out-of-the-box representations. We
argue that it is due to the fact that they only capture pixel-level knowledge
rather than spatiotemporal semantics, which hinders further progress in video
understanding. Inspired by the great success of image-text pre-training (e.g.,
CLIP), we take the first step to exploit language semantics to boost
transferable spatiotemporal representation learning. We introduce a new pretext
task, Turning to Video for Transcript Sorting (TVTS), which sorts shuffled ASR
scripts by attending to learned video representations. We do not rely on
descriptive captions and learn purely from video, i.e., leveraging the natural
transcribed speech knowledge to provide noisy but useful semantics over time.
Our method enforces the vision model to contextualize what is happening over
time so that it can re-organize the narrative transcripts, and can seamlessly
apply to large-scale uncurated video data in the real world. Our method
demonstrates strong out-of-the-box spatiotemporal representations on diverse
benchmarks, e.g., +13.6% gains over VideoMAE on SSV2 via linear probing. The
code is available at https://github.com/TencentARC/TVTS.
