  Probability predictions from binary regressions or machine learning methods
ought to be calibrated: If an event is predicted to occur with probability $x$,
it should materialize with approximately that frequency, which means that the
so-called calibration curve $p(\cdot)$ should equal the identity, $p(x) = x$
for all $x$ in the unit interval. We propose honest calibration assessment
based on novel confidence bands for the calibration curve, which are valid only
subject to the natural assumption of isotonicity. Besides testing the classical
goodness-of-fit null hypothesis of perfect calibration, our bands facilitate
inverted goodness-of-fit tests whose rejection allows for the sought-after
conclusion of a sufficiently well specified model. We show that our bands have
a finite sample coverage guarantee, are narrower than existing approaches, and
adapt to the local smoothness of the calibration curve $p$ and the local
variance of the binary observations. In an application to model predictions of
an infant having a low birth weight, the bounds give informative insights on
model calibration.
