  In this paper, we propose a novel lip-to-speech generative adversarial
network, Visual Context Attentional GAN (VCA-GAN), which can jointly model
local and global lip movements during speech synthesis. Specifically, the
proposed VCA-GAN synthesizes the speech from local lip visual features by
finding a mapping function of viseme-to-phoneme, while global visual context is
embedded into the intermediate layers of the generator to clarify the ambiguity
in the mapping induced by homophene. To achieve this, a visual context
attention module is proposed where it encodes global representations from the
local visual features, and provides the desired global visual context
corresponding to the given coarse speech representation to the generator
through audio-visual attention. In addition to the explicit modelling of local
and global visual representations, synchronization learning is introduced as a
form of contrastive learning that guides the generator to synthesize a speech
in sync with the given input lip movements. Extensive experiments demonstrate
that the proposed VCA-GAN outperforms existing state-of-the-art and is able to
effectively synthesize the speech from multi-speaker that has been barely
handled in the previous works.
