  Generating expressive and contextually appropriate prosody remains a
challenge for modern text-to-speech (TTS) systems. This is particularly evident
for long, multi-sentence inputs. In this paper, we examine simple extensions to
a Transformer-based FastSpeech-like system, with the goal of improving prosody
for multi-sentence TTS. We find that long context, powerful text features, and
training on multi-speaker data all improve prosody. More interestingly, they
result in synergies. Long context disambiguates prosody, improves coherence,
and plays to the strengths of Transformers. Fine-tuning word-level features
from a powerful language model, such as BERT, appears to profit from more
training data, readily available in a multi-speaker setting. We look into
objective metrics on pausing and pacing and perform thorough subjective
evaluations for speech naturalness. Our main system, which incorporates all the
extensions, achieves consistently strong results, including statistically
significant improvements in speech naturalness over all its competitors.
