  This paper presents a novel framework to build a voice conversion (VC) system
by learning from a text-to-speech (TTS) synthesis system, that is called TTS-VC
transfer learning. We first develop a multi-speaker speech synthesis system
with sequence-to-sequence encoder-decoder architecture, where the encoder
extracts robust linguistic representations of text, and the decoder,
conditioned on target speaker embedding, takes the context vectors and the
attention recurrent network cell output to generate target acoustic features.
We take advantage of the fact that TTS system maps input text to speaker
independent context vectors, and reuse such a mapping to supervise the training
of latent representations of an encoder-decoder voice conversion system. In the
voice conversion system, the encoder takes speech instead of text as input,
while the decoder is functionally similar to TTS decoder. As we condition the
decoder on speaker embedding, the system can be trained on non-parallel data
for any-to-any voice conversion. During voice conversion training, we present
both text and speech to speech synthesis and voice conversion networks
respectively. At run-time, the voice conversion network uses its own
encoder-decoder architecture. Experiments show that the proposed approach
outperforms two competitive voice conversion baselines consistently, namely
phonetic posteriorgram and variational autoencoder methods, in terms of speech
quality, naturalness, and speaker similarity.
