  Non-parallel many-to-many voice conversion is recently attract-ing huge
research efforts in the speech processing community. A voice conversion system
transforms an utterance of a source speaker to another utterance of a target
speaker by keeping the content in the original utterance and replacing by the
vocal features from the target speaker. Existing solutions, e.g., StarGAN-VC2,
present promising results, only when speech corpus of the engaged speakers is
available during model training. AUTOVCis able to perform voice conversion on
unseen speakers, but it needs an external pretrained speaker verification
model. In this paper, we present our new GAN-based zero-shot voice conversion
solution, called GAZEV, which targets to support unseen speakers on both source
and target utterances. Our key technical contribution is the adoption of
speaker embedding loss on top of the GAN framework, as well as adaptive
instance normalization strategy, in order to address the limitations of speaker
identity transfer in existing solutions. Our empirical evaluations demonstrate
significant performance improvement on output speech quality and comparable
speaker similarity to AUTOVC.
