  VoiceFilter-Lite is a speaker-conditioned voice separation model that plays a
crucial role in improving speech recognition and speaker verification by
suppressing overlapping speech from non-target speakers. However, one
limitation of VoiceFilter-Lite, and other speaker-conditioned speech models in
general, is that these models are usually limited to a single target speaker.
This is undesirable as most smart home devices now support multiple enrolled
users. In order to extend the benefits of personalization to multiple users, we
previously developed an attention-based speaker selection mechanism and applied
it to VoiceFilter-Lite. However, the original multi-user VoiceFilter-Lite model
suffers from significant performance degradation compared with single-user
models. In this paper, we devised a series of experiments to improve the
multi-user VoiceFilter-Lite model. By incorporating a dual learning rate
schedule and by using feature-wise linear modulation (FiLM) to condition the
model with the attended speaker embedding, we successfully closed the
performance gap between multi-user and single-user VoiceFilter-Lite models on
single-speaker evaluations. At the same time, the new model can also be easily
extended to support any number of users, and significantly outperforms our
previously published model on multi-speaker evaluations.
