  This paper introduces Taco-VC, a novel architecture for voice conversion
based on Tacotron synthesizer, which is a sequence-to-sequence with attention
model. The training of multi-speaker voice conversion systems requires a large
number of resources, both in training and corpus size. Taco-VC is implemented
using a single speaker Tacotron synthesizer based on Phonetic PosteriorGrams
(PPGs) and a single speaker WaveNet vocoder conditioned on mel spectrograms. To
enhance the converted speech quality, and to overcome over-smoothing, the
outputs of Tacotron are passed through a novel speechenhancement network, which
is composed of a combination of the phoneme recognition and Tacotron networks.
Our system is trained just with a single speaker corpus and adapts to new
speakers using only a few minutes of training data. Using mid-size public
datasets, our method outperforms the baseline in the VCC 2018 SPOKE
non-parallel voice conversion task and achieves competitive results compared to
multi-speaker networks trained on large private datasets.
