  Attention-based seq2seq text-to-speech systems, especially those use
self-attention networks (SAN), have achieved state-of-art performance. But an
expressive corpus with rich prosody is still challenging to model as 1)
prosodic aspects, which span across different sentential granularities and
mainly determine acoustic expressiveness, are difficult to quantize and label
and 2) the current seq2seq framework extracts prosodic information solely from
a text encoder, which is easily collapsed to an averaged expression for
expressive contents. In this paper, we propose a context extractor, which is
built upon SAN-based text encoder, to sufficiently exploit the sentential
context over an expressive corpus for seq2seq-based TTS. Our context extractor
first collects prosodic-related sentential context information from different
SAN layers and then aggregates them to learn a comprehensive sentence
representation to enhance the expressiveness of the final generated speech.
Specifically, we investigate two methods of context aggregation: 1) direct
aggregation which directly concatenates the outputs of different SAN layers,
and 2) weighted aggregation which uses multi-head attention to automatically
learn contributions for different SAN layers. Experiments on two expressive
corpora show that our approach can produce more natural speech with much richer
prosodic variations, and weighted aggregation is more superior in modeling
expressivity.
