  We present RAVEn, a self-supervised multi-modal approach to jointly learn
visual and auditory speech representations. Our pre-training objective involves
encoding masked inputs, and then predicting contextualised targets generated by
slowly-evolving momentum encoders. Driven by the inherent differences between
video and audio, our design is asymmetric w.r.t. the two modalities' pretext
tasks: Whereas the auditory stream predicts both the visual and auditory
targets, the visual one predicts only the auditory targets. We observe strong
results in low- and high-resource labelled data settings when fine-tuning the
visual and auditory encoders resulting from a single pre-training stage, in
which the encoders are jointly trained. Notably, RAVEn surpasses all
self-supervised methods on visual speech recognition (VSR) on LRS3, and
combining RAVEn with self-training using only 30 hours of labelled data even
outperforms a recent semi-supervised method trained on 90,000 hours of
non-public data. At the same time, we achieve state-of-the-art results in the
LRS3 low-resource setting for auditory speech recognition (as well as for VSR).
Our findings point to the viability of learning powerful speech representations
entirely from raw video and audio, i.e., without relying on handcrafted
features. Code and models are available at https://github.com/ahaliassos/raven.
