  Cross-device federated learning (FL) protects user privacy by collaboratively
training a model on user devices, therefore eliminating the need for
collecting, storing, and manually labeling user data. While important topics
such as the FL training algorithm, non-IID-ness, and Differential Privacy have
been well studied in the literature, this paper focuses on two challenges of
practical importance for improving on-device ASR: the lack of ground-truth
transcriptions and the scarcity of compute resource and network bandwidth on
edge devices. First, we propose a FL system for on-device ASR domain adaptation
with full self-supervision, which uses self-labeling together with data
augmentation and filtering techniques. The system can improve a strong
Emformer-Transducer based ASR model pretrained on out-of-domain data, using
in-domain audio without any ground-truth transcriptions. Second, to reduce the
training cost, we propose a self-restricted RNN Transducer (SR-RNN-T) loss, a
variant of alignment-restricted RNN-T that uses Viterbi alignments from
self-supervision. To further reduce the compute and network cost, we
systematically explore adapting only a subset of weights in the
Emformer-Transducer. Our best training recipe achieves a $12.9\%$ relative WER
reduction over the strong out-of-domain baseline, which equals $70\%$ of the
reduction achievable with full human supervision and centralized training.
