  Speech self-supervised models such as wav2vec 2.0 and HuBERT are making
revolutionary progress in Automatic Speech Recognition (ASR). However, they
have not been totally proven to produce better performance on tasks other than
ASR. In this work, we explored partial fine-tuning and entire fine-tuning on
wav2vec 2.0 and HuBERT pre-trained models for three non-ASR speech tasks:
Speech Emotion Recognition, Speaker Verification and Spoken Language
Understanding. With simple proposed downstream frameworks, the best scores
reached 79.58% weighted accuracy on speaker-dependent setting and 73.01%
weighted accuracy on speaker-independent setting for Speech Emotion Recognition
on IEMOCAP, 2.36% equal error rate for Speaker Verification on VoxCeleb1,
89.38% accuracy for Intent Classification and 78.92% F1 for Slot Filling on
SLURP, showing the strength of fine-tuned wav2vec 2.0 and HuBERT on learning
prosodic, voice-print and semantic representations.
