  Cued Speech (CS) is a visual coding tool to encode spoken languages at the
phonetic level, which combines lip-reading and hand gestures to effectively
assist communication among people with hearing impairments. The Automatic CS
Recognition (ACSR) task aims to recognize CS videos into linguistic texts,
which involves both lips and hands as two distinct modalities conveying
complementary information. However, the traditional centralized training
approach poses potential privacy risks due to the use of facial and gesture
videos in CS data. To address this issue, we propose a new Federated Cued
Speech Recognition (FedCSR) framework to train an ACSR model over the
decentralized CS data without sharing private information. In particular, a
mutual knowledge distillation method is proposed to maintain cross-modal
semantic consistency of the Non-IID CS data, which ensures learning a unified
feature space for both linguistic and visual information. On the server side, a
globally shared linguistic model is trained to capture the long-term
dependencies in the text sentences, which is aligned with the visual
information from the local clients via visual-to-linguistic distillation. On
the client side, the visual model of each client is trained with its own local
data, assisted by linguistic-to-visual distillation treating the linguistic
model as the teacher. To the best of our knowledge, this is the first approach
to consider the federated ACSR task for privacy protection. Experimental
results on the Chinese CS dataset with multiple cuers demonstrate that our
approach outperforms both mainstream federated learning baselines and existing
centralized state-of-the-art ACSR methods, achieving 9.7% performance
improvement for character error rate (CER) and 15.0% for word error rate (WER).
