  Lipreading or visually recognizing speech from the mouth movements of a
speaker is a challenging and mentally taxing task. Unfortunately, multiple
medical conditions force people to depend on this skill in their day-to-day
lives for essential communication. Patients suffering from Amyotrophic Lateral
Sclerosis (ALS) often lose muscle control, consequently their ability to
generate speech and communicate via lip movements. Existing large datasets do
not focus on medical patients or curate personalized vocabulary relevant to an
individual. Collecting a large-scale dataset of a patient, needed to train
mod-ern data-hungry deep learning models is, however, extremely challenging. In
this work, we propose a personalized network to lipread an ALS patient using
only one-shot examples. We depend on synthetically generated lip movements to
augment the one-shot scenario. A Variational Encoder based domain adaptation
technique is used to bridge the real-synthetic domain gap. Our approach
significantly improves and achieves high top-5accuracy with 83.2% accuracy
compared to 62.6% achieved by comparable methods for the patient. Apart from
evaluating our approach on the ALS patient, we also extend it to people with
hearing impairment relying extensively on lip movements to communicate.
