  In this paper, we propose a solution to allow speaker conditioned speech
models, such as VoiceFilter-Lite, to support an arbitrary number of enrolled
users in a single pass. This is achieved by using an attention mechanism on
multiple speaker embeddings to compute a single attentive embedding, which is
then used as a side input to the model. We implemented multi-user
VoiceFilter-Lite and evaluated it for three tasks: (1) a streaming automatic
speech recognition (ASR) task; (2) a text-independent speaker verification
task; and (3) a personalized keyphrase detection task, where ASR has to detect
keyphrases from multiple enrolled users in a noisy environment. Our experiments
show that, with up to four enrolled users, multi-user VoiceFilter-Lite is able
to significantly reduce speech recognition and speaker verification errors when
there is overlapping speech, without affecting performance under other acoustic
conditions. This attentive speaker embedding approach can also be easily
applied to other speaker-conditioned models such as personal VAD and
personalized ASR.
