  Modelling prosody variation is critical for synthesizing natural and
expressive speech in end-to-end text-to-speech (TTS) systems. In this paper, a
cross-utterance conditional VAE (CUC-VAE) is proposed to estimate a posterior
probability distribution of the latent prosody features for each phoneme by
conditioning on acoustic features, speaker information, and text features
obtained from both past and future sentences. At inference time, instead of the
standard Gaussian distribution used by VAE, CUC-VAE allows sampling from an
utterance-specific prior distribution conditioned on cross-utterance
information, which allows the prosody features generated by the TTS system to
be related to the context and is more similar to how humans naturally produce
prosody. The performance of CUC-VAE is evaluated via a qualitative listening
test for naturalness, intelligibility and quantitative measurements, including
word error rates and the standard deviation of prosody attributes. Experimental
results on LJ-Speech and LibriTTS data show that the proposed CUC-VAE TTS
system improves naturalness and prosody diversity with clear margins.
