  Model calibration aims to align confidence with prediction correctness. The
Cross-Entropy (CE) loss is widely used for calibrator training, which enforces
the model to increase confidence on the ground truth class. However, we find
the CE loss has intrinsic limitations. For example, for a narrow
misclassification, a calibrator trained by the CE loss often produces high
confidence on the wrongly predicted class (e.g., a test sample is wrongly
classified and its softmax score on the ground truth class is around 0.4),
which is undesirable. In this paper, we propose a new post-hoc calibration
objective derived from the aim of calibration. Intuitively, the proposed
objective function asks that the calibrator decrease model confidence on
wrongly predicted samples and increase confidence on correctly predicted
samples. Because a sample itself has insufficient ability to indicate
correctness, we use its transformed versions (e.g., rotated, greyscaled and
color-jittered) during calibrator training. Trained on an in-distribution
validation set and tested with isolated, individual test samples, our method
achieves competitive calibration performance on both in-distribution and
out-of-distribution test sets compared with the state of the art. Further, our
analysis points out the difference between our method and commonly used
objectives such as CE loss and mean square error loss, where the latters
sometimes deviates from the calibration aim.
