  Sequence-to-sequence (seq2seq) voice conversion (VC) models are attractive
owing to their ability to convert prosody. Nonetheless, without sufficient
data, seq2seq VC models can suffer from unstable training and mispronunciation
problems in the converted speech, thus far from practical. To tackle these
shortcomings, we propose to transfer knowledge from other speech processing
tasks where large-scale corpora are easily available, typically text-to-speech
(TTS) and automatic speech recognition (ASR). We argue that VC models
initialized with such pretrained ASR or TTS model parameters can generate
effective hidden representations for high-fidelity, highly intelligible
converted speech. We apply such techniques to recurrent neural network
(RNN)-based and Transformer based models, and through systematical experiments,
we demonstrate the effectiveness of the pretraining scheme and the superiority
of Transformer based models over RNN-based models in terms of intelligibility,
naturalness, and similarity.
