  Understanding the relationship between the auditory and visual signals is
crucial for many different applications ranging from computer-generated imagery
(CGI) and video editing automation to assisting people with hearing or visual
impairments. However, this is challenging since the distribution of both audio
and visual modality is inherently multimodal. Therefore, most of the existing
methods ignore the multimodal aspect and assume that there only exists a
deterministic one-to-one mapping between the two modalities. It can lead to
low-quality predictions as the model collapses to optimizing the average
behavior rather than learning the full data distributions. In this paper, we
present a stochastic model for generating speech in a silent video. The
proposed model combines recurrent neural networks and variational deep
generative models to learn the auditory signal's conditional distribution given
the visual signal. We demonstrate the performance of our model on the GRID
dataset based on standard benchmarks.
