  In this paper, we formulate a novel task to synthesize speech in sync with a
silent pre-recorded video, denoted as automatic voice over (AVO). Unlike
traditional speech synthesis, AVO seeks to generate not only human-sounding
speech, but also perfect lip-speech synchronization. A natural solution to AVO
is to condition the speech rendering on the temporal progression of lip
sequence in the video. We propose a novel text-to-speech model that is
conditioned on visual input, named VisualTTS, for accurate lip-speech
synchronization. The proposed VisualTTS adopts two novel mechanisms that are 1)
textual-visual attention, and 2) visual fusion strategy during acoustic
decoding, which both contribute to forming accurate alignment between the input
text content and lip motion in input lip sequence. Experimental results show
that VisualTTS achieves accurate lip-speech synchronization and outperforms all
baseline systems.
