  Lip reading has witnessed unparalleled development in recent years thanks to
deep learning and the availability of large-scale datasets. Despite the
encouraging results achieved, the performance of lip reading, unfortunately,
remains inferior to the one of its counterpart speech recognition, due to the
ambiguous nature of its actuations that makes it challenging to extract
discriminant features from the lip movement videos. In this paper, we propose a
new method, termed as Lip by Speech (LIBS), of which the goal is to strengthen
lip reading by learning from speech recognizers. The rationale behind our
approach is that the features extracted from speech recognizers may provide
complementary and discriminant clues, which are formidable to be obtained from
the subtle movements of the lips, and consequently facilitate the training of
lip readers. This is achieved, specifically, by distilling multi-granularity
knowledge from speech recognizers to lip readers. To conduct this cross-modal
knowledge distillation, we utilize an efficacious alignment scheme to handle
the inconsistent lengths of the audios and videos, as well as an innovative
filtering strategy to refine the speech recognizer's prediction. The proposed
method achieves the new state-of-the-art performance on the CMLR and LRS2
datasets, outperforming the baseline by a margin of 7.66% and 2.75% in
character error rate, respectively.
