  Pre-training on large scale unlabelled datasets has shown impressive
performance improvements in the fields of computer vision and natural language
processing. Given the advent of large-scale instructional video datasets, a
common strategy for pre-training video encoders is to use the accompanying
speech as weak supervision. However, as speech is used to supervise the
pre-training, it is never seen by the video encoder, which does not learn to
process that modality. We address this drawback of current pre-training
methods, which fail to exploit the rich cues in spoken language. Our proposal
is to pre-train a video encoder using all the available video modalities as
supervision, namely, appearance, sound, and transcribed speech. We mask an
entire modality in the input and predict it using the other two modalities.
This encourages each modality to collaborate with the others, and our video
encoder learns to process appearance and audio as well as speech. We show the
superior performance of our "modality masking" pre-training approach for video
retrieval on the How2R, YouCook2 and Condensed Movies datasets.
