  Recently reported state-of-the-art results in visual speech recognition (VSR)
often rely on increasingly large amounts of video data, while the publicly
available transcribed video datasets are limited in size. In this paper, for
the first time, we study the potential of leveraging synthetic visual data for
VSR. Our method, termed SynthVSR, substantially improves the performance of VSR
systems with synthetic lip movements. The key idea behind SynthVSR is to
leverage a speech-driven lip animation model that generates lip movements
conditioned on the input speech. The speech-driven lip animation model is
trained on an unlabeled audio-visual dataset and could be further optimized
towards a pre-trained VSR model when labeled videos are available. As plenty of
transcribed acoustic data and face images are available, we are able to
generate large-scale synthetic data using the proposed lip animation model for
semi-supervised VSR training. We evaluate the performance of our approach on
the largest public VSR benchmark - Lip Reading Sentences 3 (LRS3). SynthVSR
achieves a WER of 43.3% with only 30 hours of real labeled data, outperforming
off-the-shelf approaches using thousands of hours of video. The WER is further
reduced to 27.9% when using all 438 hours of labeled data from LRS3, which is
on par with the state-of-the-art self-supervised AV-HuBERT method. Furthermore,
when combined with large-scale pseudo-labeled audio-visual data SynthVSR yields
a new state-of-the-art VSR WER of 16.9% using publicly available data only,
surpassing the recent state-of-the-art approaches trained with 29 times more
non-public machine-transcribed video data (90,000 hours). Finally, we perform
extensive ablation studies to understand the effect of each component in our
proposed method.
