  Calibration measures and reliability diagrams are two fundamental tools for
measuring and interpreting the calibration of probabilistic predictors.
Calibration measures quantify the degree of miscalibration, and reliability
diagrams visualize the structure of this miscalibration. However, the most
common constructions of reliability diagrams and calibration measures --
binning and ECE -- both suffer from well-known flaws (e.g. discontinuity). We
show that a simple modification fixes both constructions: first smooth the
observations using an RBF kernel, then compute the Expected Calibration Error
(ECE) of this smoothed function. We prove that with a careful choice of
bandwidth, this method yields a calibration measure that is well-behaved in the
sense of (B{\l}asiok, Gopalan, Hu, and Nakkiran 2023a) -- a consistent
calibration measure. We call this measure the SmoothECE. Moreover, the
reliability diagram obtained from this smoothed function visually encodes the
SmoothECE, just as binned reliability diagrams encode the BinnedECE.
  We also provide a Python package with simple, hyperparameter-free methods for
measuring and plotting calibration: `pip install relplot\`.
