  A machine learning model is calibrated if its predicted probability for an
outcome matches the observed frequency for that outcome conditional on the
model prediction. This property has become increasingly important as the impact
of machine learning models has continued to spread to various domains. As a
result, there are now a dizzying number of recent papers on measuring and
improving the calibration of (specifically deep learning) models. In this work,
we reassess the reporting of calibration metrics in the recent literature. We
show that there exist trivial recalibration approaches that can appear
seemingly state-of-the-art unless calibration and prediction metrics (i.e. test
accuracy) are accompanied by additional generalization metrics such as negative
log-likelihood. We then derive a calibration-based decomposition of Bregman
divergences that can be used to both motivate a choice of calibration metric
based on a generalization metric, and to detect trivial calibration. Finally,
we apply these ideas to develop a new extension to reliability diagrams that
can be used to jointly visualize calibration as well as the estimated
generalization error of a model.
