  An automatic speaker verification system aims to verify the speaker identity
of a speech signal. However, a voice conversion system could manipulate a
person's speech signal to make it sound like another speaker's voice and
deceive the speaker verification system. Most countermeasures for voice
conversion-based spoofing attacks are designed to discriminate bona fide speech
from spoofed speech for speaker verification systems. In this paper, we
investigate the problem of source speaker identification -- inferring the
identity of the source speaker given the voice converted speech. To perform
source speaker identification, we simply add voice-converted speech data with
the label of source speaker identity to the genuine speech dataset during
speaker embedding network training. Experimental results show the feasibility
of source speaker identification when training and testing with converted
speeches from the same voice conversion model(s). In addition, our results
demonstrate that having more converted utterances from various voice conversion
model for training helps improve the source speaker identification performance
on converted utterances from unseen voice conversion models.
