  In this paper, a new learning algorithm for Federated Learning (FL) is
introduced. The proposed scheme is based on a weighted gradient aggregation
using two-step optimization to offer a flexible training pipeline. Herein, two
different flavors of the aggregation method are presented, leading to an order
of magnitude improvement in convergence speed compared to other distributed or
FL training algorithms like BMUF and FedAvg. Further, the aggregation algorithm
acts as a regularizer of the gradient quality. We investigate the effect of our
FL algorithm in supervised and unsupervised Speech Recognition (SR) scenarios.
The experimental validation is performed based on three tasks: first, the
LibriSpeech task showing a speed-up of 7x and 6% word error rate reduction
(WERR) compared to the baseline results. The second task is based on session
adaptation providing 20% WERR over a powerful LAS model. Finally, our
unsupervised pipeline is applied to the conversational SR task. The proposed FL
system outperforms the baseline systems in both convergence speed and overall
model performance.
