  Emergence of artificial intelligence techniques in biomedical applications
urges the researchers to pay more attention on the uncertainty quantification
(UQ) in machine-assisted medical decision making. For classification tasks,
prior studies on UQ are difficult to compare with each other, due to the lack
of a unified quantitative evaluation metric. Considering that well-performing
UQ models ought to know when the classification models act incorrectly, we
design a new evaluation metric, area under Confidence-Classification
Characteristic curves (AUCCC), to quantitatively evaluate the performance of
the UQ models. AUCCC is threshold-free, robust to perturbation, and insensitive
to the classification performance. We evaluate several UQ methods (e.g., max
softmax output) with AUCCC to validate its effectiveness. Furthermore, a simple
scheme, named Uncertainty Distillation (UDist), is developed to boost the UQ
performance, where a confidence model is distilling the confidence estimated by
deep ensembles. The proposed method is easy to implement; it consistently
outperforms strong baselines on natural and medical image datasets in our
experiments.
