  In computational histopathology algorithms now outperform humans on a range
of tasks, but to date none are employed for automated diagnoses in the clinic.
Before algorithms can be involved in such high-stakes decisions they need to
"know when they don't know", i.e., they need to estimate their predictive
uncertainty. This allows them to defer potentially erroneous predictions to a
human pathologist, thus increasing their safety. Here, we evaluate the
predictive performance and calibration of several uncertainty estimation
methods on clinical histopathology data. We show that a distance-aware
uncertainty estimation method outperforms commonly used approaches, such as
Monte Carlo dropout and deep ensembles. However, we observe a drop in
predictive performance and calibration on novel samples across all uncertainty
estimation methods tested. We also investigate the use of uncertainty
thresholding to reject out-of-distribution samples for selective prediction. We
demonstrate the limitations of this approach and suggest areas for future
research.
