  Cued Speech (CS) is a visual communication system for the deaf or hearing
impaired people. It combines lip movements with hand cues to obtain a complete
phonetic repertoire. Current deep learning based methods on automatic CS
recognition suffer from a common problem, which is the data scarcity. Until
now, there are only two public single speaker datasets for French (238
sentences) and British English (97 sentences). In this work, we propose a
cross-modal knowledge distillation method with teacher-student structure, which
transfers audio speech information to CS to overcome the limited data problem.
Firstly, we pretrain a teacher model for CS recognition with a large amount of
open source audio speech data, and simultaneously pretrain the feature
extractors for lips and hands using CS data. Then, we distill the knowledge
from teacher model to the student model with frame-level and sequence-level
distillation strategies. Importantly, for frame-level, we exploit multi-task
learning to weigh losses automatically, to obtain the balance coefficient.
Besides, we establish a five-speaker British English CS dataset for the first
time. The proposed method is evaluated on French and British English CS
datasets, showing superior CS recognition performance to the state-of-the-art
(SOTA) by a large margin.
