  Binary classification is highly used in credit scoring in the estimation of
probability of default. The validation of such predictive models is based both
on rank ability, and also on calibration (i.e. how accurately the probabilities
output by the model map to the observed probabilities). In this study we cover
the current best practices regarding calibration for binary classification, and
explore how different approaches yield different results on real world credit
scoring data. The limitations of evaluating credit scoring models using only
rank ability metrics are explored. A benchmark is run on 18 real world
datasets, and results compared. The calibration techniques used are Platt
Scaling and Isotonic Regression. Also, different machine learning models are
used: Logistic Regression, Random Forest Classifiers, and Gradient Boosting
Classifiers. Results show that when the dataset is treated as a time series,
the use of re-calibration with Isotonic Regression is able to improve the long
term calibration better than the alternative methods. Using re-calibration, the
non-parametric models are able to outperform the Logistic Regression on Brier
Score Loss.
