  Talking face generation, also known as speech-to-lip generation, reconstructs
facial motions concerning lips given coherent speech input. The previous
studies revealed the importance of lip-speech synchronization and visual
quality. Despite much progress, they hardly focus on the content of lip
movements i.e., the visual intelligibility of the spoken words, which is an
important aspect of generation quality. To address the problem, we propose
using a lip-reading expert to improve the intelligibility of the generated lip
regions by penalizing the incorrect generation results. Moreover, to compensate
for data scarcity, we train the lip-reading expert in an audio-visual
self-supervised manner. With a lip-reading expert, we propose a novel
contrastive learning to enhance lip-speech synchronization, and a transformer
to encode audio synchronically with video, while considering global temporal
dependency of audio. For evaluation, we propose a new strategy with two
different lip-reading experts to measure intelligibility of the generated
videos. Rigorous experiments show that our proposal is superior to other
State-of-the-art (SOTA) methods, such as Wav2Lip, in reading intelligibility
i.e., over 38% Word Error Rate (WER) on LRS2 dataset and 27.8% accuracy on LRW
dataset. We also achieve the SOTA performance in lip-speech synchronization and
comparable performances in visual quality.
