  This paper presents a novel metric learning approach to address the
performance gap between normal and silent speech in visual speech recognition
(VSR). The difference in lip movements between the two poses a challenge for
existing VSR models, which exhibit degraded accuracy when applied to silent
speech. To solve this issue and tackle the scarcity of training data for silent
speech, we propose to leverage the shared literal content between normal and
silent speech and present a metric learning approach based on visemes.
Specifically, we aim to map the input of two speech types close to each other
in a latent space if they have similar viseme representations. By minimizing
the Kullback-Leibler divergence of the predicted viseme probability
distributions between and within the two speech types, our model effectively
learns and predicts viseme identities. Our evaluation demonstrates that our
method improves the accuracy of silent VSR, even when limited training data is
available.
