  Large, pre-trained neural networks consisting of self-attention layers
(transformers) have recently achieved state-of-the-art results on several
speech emotion recognition (SER) datasets. These models are typically
pre-trained in self-supervised manner with the goal to improve automatic speech
recognition performance -- and thus, to understand linguistic information. In
this work, we investigate the extent in which this information is exploited
during SER fine-tuning. Using a reproducible methodology based on open-source
tools, we synthesise prosodically neutral speech utterances while varying the
sentiment of the text. Valence predictions of the transformer model are very
reactive to positive and negative sentiment content, as well as negations, but
not to intensifiers or reducers, while none of those linguistic features impact
arousal or dominance. These findings show that transformers can successfully
leverage linguistic information to improve their valence predictions, and that
linguistic analysis should be included in their testing.
