  For users to trust model predictions, they need to understand model outputs,
particularly their confidence - calibration aims to adjust (calibrate) models'
confidence to match expected accuracy. We argue that the traditional
calibration evaluation does not promote effective calibrations: for example, it
can encourage always assigning a mediocre confidence score to all predictions,
which does not help users distinguish correct predictions from wrong ones.
Building on those observations, we propose a new calibration metric, MacroCE,
that better captures whether the model assigns low confidence to wrong
predictions and high confidence to correct predictions. Focusing on the
practical application of open-domain question answering, we examine
conventional calibration methods applied on the widely-used retriever-reader
pipeline, all of which do not bring significant gains under our new MacroCE
metric. Toward better calibration, we propose a new calibration method
(ConsCal) that uses not just final model predictions but whether multiple model
checkpoints make consistent predictions. Altogether, we provide an alternative
view of calibration along with a new metric, re-evaluation of existing
calibration methods on our metric, and proposal of a more effective calibration
method.
