  In large part due to their implicit semantic modeling, self-supervised
learning (SSL) methods have significantly increased the performance of valence
recognition in speech emotion recognition (SER) systems. Yet, their large size
may often hinder practical implementations. In this work, we take HuBERT as an
example of an SSL model and analyze the relevance of each of its layers for
SER. We show that shallow layers are more important for arousal recognition
while deeper layers are more important for valence. This observation motivates
the importance of additional textual information for accurate valence
recognition, as the distilled framework lacks the depth of its large-scale SSL
teacher. Thus, we propose an audio-textual distilled SSL framework that, while
having only ~20% of the trainable parameters of a large SSL model, achieves on
par performance across the three emotion dimensions (arousal, valence,
dominance) on the MSP-Podcast v1.10 dataset.
