  The assessment of binary classifier performance traditionally centers on
discriminative ability using metrics, such as accuracy. However, these metrics
often disregard the model's inherent uncertainty, especially when dealing with
sensitive decision-making domains, such as finance or healthcare. Given that
model-predicted scores are commonly seen as event probabilities, calibration is
crucial for accurate interpretation. In our study, we analyze the sensitivity
of various calibration measures to score distortions and introduce a refined
metric, the Local Calibration Score. Comparing recalibration methods, we
advocate for local regressions, emphasizing their dual role as effective
recalibration tools and facilitators of smoother visualizations. We apply these
findings in a real-world scenario using Random Forest classifier and regressor
to predict credit default while simultaneously measuring calibration during
performance optimization.
