  Generic pre-trained speech and text representations promise to reduce the
need for large labeled datasets on specific speech and language tasks. However,
it is not clear how to effectively adapt these representations for speech
emotion recognition. Recent public benchmarks show the efficacy of several
popular self-supervised speech representations for emotion classification. In
this study, we show that the primary difference between the top-performing
representations is in predicting valence while the differences in predicting
activation and dominance dimensions are less pronounced. However, we show that
even the best-performing HuBERT representation underperforms on valence
prediction compared to a multimodal model that also incorporates text
representation. We address this shortcoming by injecting lexical information
into the speech representation using the multimodal model as a teacher. To
improve the efficacy of our approach, we propose a novel estimate of the
quality of the emotion predictions, to condition teacher-student training. We
report new audio-only state-of-the-art concordance correlation coefficient
(CCC) values of 0.757, 0.627, 0.671 for activation, valence and dominance
predictions, respectively, on the MSP-Podcast corpus, and also state-of-the-art
values of 0.667, 0.582, 0.545 on the IEMOCAP corpus.
