  Machine learning-based User Authentication (UA) models have been widely
deployed in smart devices. UA models are trained to map input data of different
users to highly separable embedding vectors, which are then used to accept or
reject new inputs at test time. Training UA models requires having direct
access to the raw inputs and embedding vectors of users, both of which are
privacy-sensitive information. In this paper, we propose Federated User
Authentication (FedUA), a framework for privacy-preserving training of UA
models. FedUA adopts federated learning framework to enable a group of users to
jointly train a model without sharing the raw inputs. It also allows users to
generate their embeddings as random binary vectors, so that, unlike the
existing approach of constructing the spread out embeddings by the server, the
embedding vectors are kept private as well. We show our method is
privacy-preserving, scalable with number of users, and allows new users to be
added to training without changing the output layer. Our experimental results
on the VoxCeleb dataset for speaker verification shows our method reliably
rejects data of unseen users at very high true positive rates.
