  Biometrics are one of the most privacy-sensitive data. Ubiquitous
authentication systems with a focus on privacy favor decentralized approaches
as they reduce potential attack vectors, both on a technical and organizational
level. The gold standard is to let the user be in control of where their own
data is stored, which consequently leads to a high variety of devices used.
Moreover, in comparison with a centralized system, designs with higher end-user
freedom often incur additional network overhead. Therefore, when using face
recognition for biometric authentication, an efficient way to compare faces is
important in practical deployments, because it reduces both network and
hardware requirements that are essential to encourage device diversity. This
paper proposes an efficient way to aggregate embeddings used for face
recognition based on an extensive analysis on different datasets and the use of
different aggregation strategies. As part of this analysis, a new dataset has
been collected, which is available for research purposes. Our proposed method
supports the construction of massively scalable, decentralized face recognition
systems with a focus on both privacy and long-term usability.
