  Speech model adaptation is crucial to handle the discrepancy between
server-side proxy training data and actual data received on local devices of
users. With the use of federated learning (FL), we introduce an efficient
approach on continuously adapting neural network language models (NNLMs) on
private devices with applications on automatic speech recognition (ASR). To
address the potential speech transcription errors in the on-device training
corpus, we perform empirical studies on comparing various strategies of
leveraging token-level confidence scores to improve the NNLM quality in the FL
settings. Experiments show that compared with no model adaptation, the proposed
method achieves relative 2.6% and 10.8% word error rate (WER) reductions on two
speech evaluation datasets, respectively. We also provide analysis in
evaluating privacy guarantees of our presented procedure.
