  Current two-stage TTS framework typically integrates an acoustic model with a
vocoder -- the acoustic model predicts a low resolution intermediate
representation such as Mel-spectrum while the vocoder generates waveform from
the intermediate representation. Although the intermediate representation is
served as a bridge, there still exists critical mismatch between the acoustic
model and the vocoder as they are commonly separately learned and work on
different distributions of representation, leading to inevitable artifacts in
the synthesized speech. In this work, different from using pre-designed
intermediate representation in most previous studies, we propose to use VAE
combining with GAN to learn a latent representation directly from speech and
then utilize a flow-based acoustic model to model the distribution of the
latent representation from text. In this way, the mismatch problem is migrated
as the two stages work on the same distribution. Results demonstrate that the
flow-based acoustic model can exactly model the distribution of our learned
speech representation and the proposed TTS framework, namely Glow-WaveGAN, can
produce high fidelity speech outperforming the state-of-the-art GAN-based
model.
