  We propose a novel training scheme to optimize voice conversion network with
a speaker identity loss function. The training scheme not only minimizes
frame-level spectral loss, but also speaker identity loss. We introduce a cycle
consistency loss that constrains the converted speech to maintain the same
speaker identity as reference speech at utterance level. While the proposed
training scheme is applicable to any voice conversion networks, we formulate
the study under the average model voice conversion framework in this paper.
Experiments conducted on CMU-ARCTIC and CSTR-VCTK corpus confirm that the
proposed method outperforms baseline methods in terms of speaker similarity.
