  For speech classification tasks, deep learning models often achieve high
accuracy but exhibit shortcomings in calibration, manifesting as classifiers
exhibiting overconfidence. The significance of calibration lies in its critical
role in guaranteeing the reliability of decision-making within deep learning
systems. This study explores the effectiveness of Energy-Based Models in
calibrating confidence for speech classification tasks by training a joint EBM
integrating a discriminative and a generative model, thereby enhancing the
classifiers calibration and mitigating overconfidence. Experimental evaluations
conducted on three speech classification tasks specifically: age, emotion, and
language recognition. Our findings highlight the competitive performance of
EBMs in calibrating the speech classification models. This research emphasizes
the potential of EBMs in speech classification tasks, demonstrating their
ability to enhance calibration without sacrificing accuracy.
