  We introduce federated marginal personalization (FMP), a novel method for
continuously updating personalized neural network language models (NNLMs) on
private devices using federated learning (FL). Instead of fine-tuning the
parameters of NNLMs on personal data, FMP regularly estimates global and
personalized marginal distributions of words, and adjusts the probabilities
from NNLMs by an adaptation factor that is specific to each word. Our presented
approach can overcome the limitations of federated fine-tuning and efficiently
learn personalized NNLMs on devices. We study the application of FMP on
second-pass ASR rescoring tasks. Experiments on two speech evaluation datasets
show modest word error rate (WER) reductions. We also demonstrate that FMP
could offer reasonable privacy with only a negligible cost in speech
recognition accuracy.
