  Emotional voice conversion aims to transform emotional prosody in speech
while preserving the linguistic content and speaker identity. Prior studies
show that it is possible to disentangle emotional prosody using an
encoder-decoder network conditioned on discrete representation, such as one-hot
emotion labels. Such networks learn to remember a fixed set of emotional
styles. In this paper, we propose a novel framework based on variational
auto-encoding Wasserstein generative adversarial network (VAW-GAN), which makes
use of a pre-trained speech emotion recognition (SER) model to transfer
emotional style during training and at run-time inference. In this way, the
network is able to transfer both seen and unseen emotional style to a new
utterance. We show that the proposed framework achieves remarkable performance
by consistently outperforming the baseline framework. This paper also marks the
release of an emotional speech dataset (ESD) for voice conversion, which has
multiple speakers and languages.
