  Embedding paralinguistic properties is a challenging task as there are only a
few hours of training data available for domains such as emotional speech. One
solution to this problem is to pretrain a general self-supervised speech
representation model on large amounts of unlabeled speech. This pretrained
model is then finetuned to a specific task. Paralinguistic properties however
have notoriously high class variance, making the finetuning ineffective. In
this work, we propose a two step approach to this. First we improve the
embedding space, then we train an adapter to bridge the gap from the embedding
space to a classification task. In order to improve the class invariance we use
a combination of contrastive and non-contrastive losses to explicitly optimize
for class invariant, yet discriminative features. Our approach consistently
outperforms baselines that are finetuned end-to-end on multiple tasks and
surpasses a benchmark on state-of-the-art emotion classification.
