  Visual cues, like lip motion, have been shown to improve the performance of
Automatic Speech Recognition (ASR) systems in noisy environments. We propose
LipGER (Lip Motion aided Generative Error Correction), a novel framework for
leveraging visual cues for noise-robust ASR. Instead of learning the
cross-modal correlation between the audio and visual modalities, we make an LLM
learn the task of visually-conditioned (generative) ASR error correction.
Specifically, we instruct an LLM to predict the transcription from the N-best
hypotheses generated using ASR beam-search. This is further conditioned on lip
motions. This approach addresses key challenges in traditional AVSR learning,
such as the lack of large-scale paired datasets and difficulties in adapting to
new domains. We experiment on 4 datasets in various settings and show that
LipGER improves the Word Error Rate in the range of 1.1%-49.2%. We also release
LipHyp, a large-scale dataset with hypothesis-transcription pairs that is
additionally equipped with lip motion cues to promote further research in this
space
