  Accurate uncertainty quantification is crucial for the safe deployment of
machine learning models, and prior research has demonstrated improvements in
the calibration of modern language models (LMs). We study in-context learning
(ICL), a prevalent method for adapting static LMs through tailored prompts, and
examine the balance between performance and calibration across a broad spectrum
of natural language understanding and reasoning tasks. Through comprehensive
experiments, we observe that, with an increasing number of ICL examples, models
initially exhibit increased miscalibration before achieving better calibration
and miscalibration tends to arise in low-shot settings. Moreover, we find that
methods aimed at improving usability, such as fine-tuning and chain-of-thought
(CoT) prompting, can lead to miscalibration and unreliable natural language
explanations. Furthermore, we explore recalibration techniques and find that a
scaling-binning calibrator can reduce calibration errors consistently.
