  In many classification problems it is desirable to output well-calibrated
probabilities on the different classes. We propose a robust, non-parametric
method of calibrating probabilities called SplineCalib that utilizes smoothing
splines to determine a calibration function. We demonstrate how applying
certain transformations as part of the calibration process can improve
performance on problems in deep learning and other domains where the scores
tend to be "overconfident". We adapt the approach to multi-class problems and
find that better calibration can improve accuracy as well as log-loss by better
resolving uncertain cases. Finally, we present a cross-validated approach to
calibration which conserves data. Significant improvements to log-loss and
accuracy are shown on several different problems. We also introduce the
ml-insights python package which contains an implementation of the SplineCalib
algorithm.
