  Humans can effortlessly modify various prosodic attributes, such as the
placement of stress and the intensity of sentiment, to convey a specific
emotion while maintaining consistent linguistic content. Motivated by this
capability, we propose EmoAug, a novel style transfer model designed to enhance
emotional expression and tackle the data scarcity issue in speech emotion
recognition tasks. EmoAug consists of a semantic encoder and a paralinguistic
encoder that represent verbal and non-verbal information respectively.
Additionally, a decoder reconstructs speech signals by conditioning on the
aforementioned two information flows in an unsupervised fashion. Once training
is completed, EmoAug enriches expressions of emotional speech with different
prosodic attributes, such as stress, rhythm and intensity, by feeding different
styles into the paralinguistic encoder. EmoAug enables us to generate similar
numbers of samples for each class to tackle the data imbalance issue as well.
Experimental results on the IEMOCAP dataset demonstrate that EmoAug can
successfully transfer different speaking styles while retaining the speaker
identity and semantic content. Furthermore, we train a SER model with data
augmented by EmoAug and show that the augmented model not only surpasses the
state-of-the-art supervised and self-supervised methods but also overcomes
overfitting problems caused by data imbalance. Some audio samples can be found
on our demo website.
