  Many speech synthesis datasets, especially those derived from audiobooks,
naturally comprise sequences of utterances. Nevertheless, such data are
commonly treated as individual, unordered utterances both when training a model
and at inference time. This discards important prosodic phenomena above the
utterance level. In this paper, we leverage the sequential nature of the data
using an acoustic context encoder that produces an embedding of the previous
utterance audio. This is input to the decoder in a Tacotron 2 model. The
embedding is also used for a secondary task, providing additional supervision.
We compare two secondary tasks: predicting the ordering of utterance pairs, and
predicting the embedding of the current utterance audio. Results show that the
relation between consecutive utterances is informative: our proposed model
significantly improves naturalness over a Tacotron 2 baseline.
