  We study the fundamental question of how to define and measure the distance
from calibration for probabilistic predictors. While the notion of perfect
calibration is well-understood, there is no consensus on how to quantify the
distance from perfect calibration. Numerous calibration measures have been
proposed in the literature, but it is unclear how they compare to each other,
and many popular measures such as Expected Calibration Error (ECE) fail to
satisfy basic properties like continuity.
  We present a rigorous framework for analyzing calibration measures, inspired
by the literature on property testing. We propose a ground-truth notion of
distance from calibration: the $\ell_1$ distance to the nearest perfectly
calibrated predictor. We define a consistent calibration measure as one that is
polynomially related to this distance. Applying our framework, we identify
three calibration measures that are consistent and can be estimated
efficiently: smooth calibration, interval calibration, and Laplace kernel
calibration. The former two give quadratic approximations to the ground truth
distance, which we show is information-theoretically optimal in a natural model
for measuring calibration which we term the prediction-only access model. Our
work thus establishes fundamental lower and upper bounds on measuring the
distance to calibration, and also provides theoretical justification for
preferring certain metrics (like Laplace kernel calibration) in practice.
