  Automatic speech recognition (ASR) models with low-footprint are increasingly
being deployed on edge devices for conversational agents, which enhances
privacy. We study the problem of federated continual incremental learning for
recurrent neural network-transducer (RNN-T) ASR models in the privacy-enhancing
scheme of learning on-device, without access to ground truth human transcripts
or machine transcriptions from a stronger ASR model. In particular, we study
the performance of a self-learning based scheme, with a paired teacher model
updated through an exponential moving average of ASR models. Further, we
propose using possibly noisy weak-supervision signals such as feedback scores
and natural language understanding semantics determined from user behavior
across multiple turns in a session of interactions with the conversational
agent. These signals are leveraged in a multi-task policy-gradient training
approach to improve the performance of self-learning for ASR. Finally, we show
how catastrophic forgetting can be mitigated by combining on-device learning
with a memory-replay approach using selected historical datasets. These
innovations allow for 10% relative improvement in WER on new use cases with
minimal degradation on other test sets in the absence of strong-supervision
signals such as ground-truth transcriptions.
