  This paper is concerned with the task of speaker verification on audio with
multiple overlapping speakers. Most speaker verification systems are designed
with the assumption of a single speaker being present in a given audio segment.
However, in a real-world setting this assumption does not always hold. In this
paper, we demonstrate that current speaker verification systems are not robust
against audio with noticeable speaker overlap. To alleviate this issue, we
propose margin-mixup, a simple training strategy that can easily be adopted by
existing speaker verification pipelines to make the resulting speaker
embeddings robust against multi-speaker audio. In contrast to other methods,
margin-mixup requires no alterations to regular speaker verification
architectures, while attaining better results. On our multi-speaker test set
based on VoxCeleb1, the proposed margin-mixup strategy improves the EER on
average with 44.4% relative to our state-of-the-art speaker verification
baseline systems.
