  Keyword spotting (KWS) aims to discriminate a specific wake-up word from
other signals precisely and efficiently for different users. Recent works
utilize various deep networks to train KWS models with all users' speech data
centralized without considering data privacy. Federated KWS (FedKWS) could
serve as a solution without directly sharing users' data. However, the small
amount of data, different user habits, and various accents could lead to fatal
problems, e.g., overfitting or weight divergence. Hence, we propose several
strategies to encourage the model not to overfit user-specific information in
FedKWS. Specifically, we first propose an adversarial learning strategy, which
updates the downloaded global model against an overfitted local model and
explicitly encourages the global model to capture user-invariant information.
Furthermore, we propose an adaptive local training strategy, letting clients
with more training data and more uniform class distributions undertake more
local update steps. Equivalently, this strategy could weaken the negative
impacts of those users whose data is less qualified. Our proposed FedKWS-UI
could explicitly and implicitly learn user-invariant information in FedKWS.
Abundant experimental results on federated Google Speech Commands verify the
effectiveness of FedKWS-UI.
