  Large pre-trained language models (PLMs) have demonstrated strong performance
on natural language understanding (NLU) tasks through fine-tuning. However,
fine-tuned models still suffer from overconfident predictions, especially in
out-of-domain settings. In this paper, we tackle the problem of calibrating
fine-tuned language models. We demonstrate that the PLMs are well-calibrated on
the masked language modeling task with robust predictive confidence under
domain shift, yet the fine-tuned models fail to retain such property due to
catastrophic forgetting, which impacts the calibration on the downstream
classification task. In light of these observations, we evaluate the
calibration of several methods that preserve pre-trained features and show that
preserving pre-trained features can improve the calibration of fine-tuned
language models. Among these methods, our proposed method that encourages the
fine-tuned model to learn generative representations with auxiliary language
modeling objective achieves competitive accuracy and the lowest expected
calibration error compared to several strong baselines under both in-domain and
out-of-domain settings on three downstream NLU tasks.
