  Learning emotion embedding from reference audio is a straightforward approach
for multi-emotion speech synthesis in encoder-decoder systems. But how to get
better emotion embedding and how to inject it into TTS acoustic model more
effectively are still under investigation. In this paper, we propose an
innovative constraint to help VAE extract emotion embedding with better cluster
cohesion. Besides, the obtained emotion embedding is used as query to aggregate
latent representations of all encoder layers via attention. Moreover, the
queries from encoder layers themselves are also helpful. Experiments prove the
proposed methods can enhance the encoding of comprehensive syntactic and
semantic information and produce more expressive emotional speech.
