  Expressive synthetic speech is essential for many human-computer interaction
and audio broadcast scenarios, and thus synthesizing expressive speech has
attracted much attention in recent years. Previous methods performed the
expressive speech synthesis either with explicit labels or with a fixed-length
style embedding extracted from reference audio, both of which can only learn an
average style and thus ignores the multi-scale nature of speech prosody. In
this paper, we propose MsEmoTTS, a multi-scale emotional speech synthesis
framework, to model the emotion from different levels. Specifically, the
proposed method is a typical attention-based sequence-to-sequence model and
with proposed three modules, including global-level emotion presenting module
(GM), utterance-level emotion presenting module (UM), and local-level emotion
presenting module (LM), to model the global emotion category, utterance-level
emotion variation, and syllable-level emotion strength, respectively. In
addition to modeling the emotion from different levels, the proposed method
also allows us to synthesize emotional speech in different ways, i.e.,
transferring the emotion from reference audio, predicting the emotion from
input text, and controlling the emotion strength manually. Extensive
experiments conducted on a Chinese emotional speech corpus demonstrate that the
proposed method outperforms the compared reference audio-based and text-based
emotional speech synthesis methods on the emotion transfer speech synthesis and
text-based emotion prediction speech synthesis respectively. Besides, the
experiments also show that the proposed method can control the emotion
expressions flexibly. Detailed analysis shows the effectiveness of each module
and the good design of the proposed method.
