  Automated emotion recognition in speech is a long-standing problem. While
early work on emotion recognition relied on hand-crafted features and simple
classifiers, the field has now embraced end-to-end feature learning and
classification using deep neural networks. In parallel to these models,
researchers have proposed several data augmentation techniques to increase the
size and variability of existing labeled datasets. Despite many seminal
contributions in the field, we still have a poor understanding of the interplay
between the network architecture and the choice of data augmentation. Moreover,
only a handful of studies demonstrate the generalizability of a particular
model across multiple datasets, which is a prerequisite for robust real-world
performance. In this paper, we conduct a comprehensive evaluation of popular
deep learning approaches for emotion recognition. To eliminate bias, we fix the
model architectures and optimization hyperparameters using the VESUS dataset
and then use repeated 5-fold cross validation to evaluate the performance on
the IEMOCAP and CREMA-D datasets. Our results demonstrate that long-range
dependencies in the speech signal are critical for emotion recognition and that
speed/rate augmentation offers the most robust performance gain across models.
