  Speaker verification has been studied mostly under the single-talker
condition. It is adversely affected in the presence of interference speakers.
Inspired by the study on target speaker extraction, e.g., SpEx, we propose a
unified speaker verification framework for both single- and multi-talker
speech, that is able to pay selective auditory attention to the target speaker.
This target speaker verification (tSV) framework jointly optimizes a speaker
attention module and a speaker representation module via multi-task learning.
We study four different target speaker embedding schemes under the tSV
framework. The experimental results show that all four target speaker embedding
schemes significantly outperform other competitive solutions for multi-talker
speech. Notably, the best tSV speaker embedding scheme achieves 76.0% and 55.3%
relative improvements over the baseline system on the WSJ0-2mix-extr and
Libri2Mix corpora in terms of equal-error-rate for 2-talker speech, while the
performance of tSV for single-talker speech is on par with that of traditional
speaker verification system, that is trained and evaluated under the same
single-talker condition.
