  Machine learning models for speech emotion recognition (SER) can be trained
for different tasks and are usually evaluated on the basis of a few available
datasets per task. Tasks could include arousal, valence, dominance, emotional
categories, or tone of voice. Those models are mainly evaluated in terms of
correlation or recall, and always show some errors in their predictions. The
errors manifest themselves in model behaviour, which can be very different
along different dimensions even if the same recall or correlation is achieved
by the model. This paper introduces a testing framework to investigate
behaviour of speech emotion recognition models, by requiring different metrics
to reach a certain threshold in order to pass a test. The test metrics can be
grouped in terms of correctness, fairness, and robustness. It further provides
a method to specify test thresholds for fairness tests automatically, based on
the used datasets, and recommendations how to select the remaining test
thresholds. Seven different transformer based models, and a baseline model are
tested for arousal, valence, dominance, and emotional categories. The test
results highlight, that models with high correlation or recall might rely on
shortcuts - such as text sentiment - to achieve this, and differ in terms of
fairness.
