  Speech Emotion Recognition (SER) has been traditionally formulated as a
classification task. However, emotions are generally a spectrum whose
distribution varies from situation to situation leading to poor Out-of-Domain
(OOD) performance. We take inspiration from statistical formulation of
Automatic Speech Recognition (ASR) and formulate the SER task as generating the
most likely sequence of text tokens to infer emotion. The formulation breaks
SER into predicting acoustic model features weighted by language model
prediction. As an instance of this approach, we present SELM, an
audio-conditioned language model for SER that predicts different emotion views.
We train SELM on curated speech emotion corpus and test it on three OOD
datasets (RAVDESS, CREMAD, IEMOCAP) not used in training. SELM achieves
significant improvements over the state-of-the-art baselines, with 17% and 7%
relative accuracy gains for RAVDESS and CREMA-D, respectively. Moreover, SELM
can further boost its performance by Few-Shot Learning using a few annotated
examples. The results highlight the effectiveness of our SER formulation,
especially to improve performance in OOD scenarios.
