  One-shot voice conversion (VC) aims to convert speech from any source speaker
to an arbitrary target speaker with only a few seconds of reference speech from
the target speaker. This relies heavily on disentangling the speaker's identity
and speech content, a task that still remains challenging. Here, we propose a
novel approach to learning disentangled speech representation by transfer
learning from style-based text-to-speech (TTS) models. With cycle consistent
and adversarial training, the style-based TTS models can perform
transcription-guided one-shot VC with high fidelity and similarity. By learning
an additional mel-spectrogram encoder through a teacher-student knowledge
transfer and novel data augmentation scheme, our approach results in
disentangled speech representation without needing the input text. The
subjective evaluation shows that our approach can significantly outperform the
previous state-of-the-art one-shot voice conversion models in both naturalness
and similarity.
