  The continuous evolution of pre-trained speech models has greatly advanced
Speech Emotion Recognition (SER). However, current research typically relies on
utterance-level emotion labels, inadequately capturing the complexity of
emotions within a single utterance. In this paper, we introduce GMP-TL, a novel
SER framework that employs gender-augmented multi-scale pseudo-label (GMP)
based transfer learning to mitigate this gap. Specifically, GMP-TL initially
uses the pre-trained HuBERT, implementing multi-task learning and multi-scale
k-means clustering to acquire frame-level GMPs. Subsequently, to fully leverage
frame-level GMPs and utterance-level emotion labels, a two-stage model
fine-tuning approach is presented to further optimize GMP-TL. Experiments on
IEMOCAP show that our GMP-TL attains a WAR of 80.0% and an UAR of 82.0%,
achieving superior performance compared to state-of-the-art unimodal SER
methods while also yielding comparable results to multimodal SER approaches.
