  This paper investigates the post-hoc calibration of confidence for
"exploratory" machine learning classification problems. The difficulty in these
problems stems from the continuing desire to push the boundaries of which
categories have enough examples to generalize from when curating datasets, and
confusion regarding the validity of those categories. We argue that for such
problems the "one-versus-all" approach (top-label calibration) must be used
rather than the "calibrate-the-full-response-matrix" approach advocated
elsewhere in the literature. We introduce and test four new algorithms designed
to handle the idiosyncrasies of category-specific confidence estimation. Chief
among these methods is the use of kernel density ratios for confidence
calibration including a novel, bulletproof algorithm for choosing the
bandwidth. We test our claims and explore the limits of calibration on a
bioinformatics application (PhANNs) as well as the classic MNIST benchmark.
Finally, our analysis argues that post-hoc calibration should always be
performed, should be based only on the test dataset, and should be
sanity-checked visually.
