  Recently, cycle-consistent adversarial network (Cycle-GAN) has been
successfully applied to voice conversion to a different speaker without
parallel data, although in those approaches an individual model is needed for
each target speaker. In this paper, we propose an adversarial learning
framework for voice conversion, with which a single model can be trained to
convert the voice to many different speakers, all without parallel data, by
separating the speaker characteristics from the linguistic content in speech
signals. An autoencoder is first trained to extract speaker-independent latent
representations and speaker embedding separately using another auxiliary
speaker classifier to regularize the latent representation. The decoder then
takes the speaker-independent latent representation and the target speaker
embedding as the input to generate the voice of the target speaker with the
linguistic content of the source utterance. The quality of decoder output is
further improved by patching with the residual signal produced by another pair
of generator and discriminator. A target speaker set size of 20 was tested in
the preliminary experiments, and very good voice quality was obtained.
Conventional voice conversion metrics are reported. We also show that the
speaker information has been properly reduced from the latent representations.
