  Speech in-painting is the task of regenerating missing audio contents using
reliable context information. Despite various recent studies in multi-modal
perception of audio in-painting, there is still a need for an effective
infusion of visual and auditory information in speech in-painting. In this
paper, we introduce a novel sequence-to-sequence model that leverages the
visual information to in-paint audio signals via an encoder-decoder
architecture. The encoder plays the role of a lip-reader for facial recordings
and the decoder takes both encoder outputs as well as the distorted audio
spectrograms to restore the original speech. Our model outperforms an
audio-only speech in-painting model and has comparable results with a recent
multi-modal speech in-painter in terms of speech quality and intelligibility
metrics for distortions of 300 ms to 1500 ms duration, which proves the
effectiveness of the introduced multi-modality in speech in-painting.
