  In this paper we present VDTTS, a Visually-Driven Text-to-Speech model.
Motivated by dubbing, VDTTS takes advantage of video frames as an additional
input alongside text, and generates speech that matches the video signal. We
demonstrate how this allows VDTTS to, unlike plain TTS models, generate speech
that not only has prosodic variations like natural pauses and pitch, but is
also synchronized to the input video. Experimentally, we show our model
produces well-synchronized outputs, approaching the video-speech
synchronization quality of the ground-truth, on several challenging benchmarks
including "in-the-wild" content from VoxCeleb2. Supplementary demo videos
demonstrating video-speech synchronization, robustness to speaker ID swapping,
and prosody, presented at the project page.
