  Uncertainty in probabilistic classifiers predictions is a key concern when
models are used to support human decision making, in broader probabilistic
pipelines or when sensitive automatic decisions have to be taken. Studies have
shown that most models are not intrinsically well calibrated, meaning that
their decision scores are not consistent with posterior probabilities. Hence
being able to calibrate these models, or enforce calibration while learning
them, has regained interest in recent literature. In this context, properly
assessing calibration is paramount to quantify new contributions tackling
calibration. However, there is room for improvement for commonly used metrics
and evaluation of calibration could benefit from deeper analyses. Thus this
paper focuses on the empirical evaluation of calibration metrics in the context
of classification. More specifically it evaluates different estimators of the
Expected Calibration Error ($ECE$), amongst which legacy estimators and some
novel ones, proposed in this paper. We build an empirical procedure to quantify
the quality of these $ECE$ estimators, and use it to decide which estimator
should be used in practice for different settings.
