  Recent advancements in Deep and Self-Supervised Learning (SSL) have led to
substantial improvements in Speech Emotion Recognition (SER) performance,
reaching unprecedented levels. However, obtaining sufficient amounts of
accurately labeled data for training or fine-tuning the models remains a costly
and challenging task. In this paper, we propose a multi-view SSL pre-training
technique that can be applied to various representations of speech, including
the ones generated by large speech models, to improve SER performance in
scenarios where annotations are limited. Our experiments, based on wav2vec 2.0,
spectral and paralinguistic features, demonstrate that the proposed framework
boosts the SER performance, by up to 10% in Unweighted Average Recall, in
settings with extremely sparse data annotations.
