  In a data-scarce field such as healthcare, where models often deliver
predictions on patients with rare conditions, the ability to measure the
uncertainty of a model's prediction could potentially lead to improved
effectiveness of decision support tools and increased user trust. This work
advances the understanding of uncertainty estimation for classification and
risk prediction on medical tabular data, in a two-fold way. First, we expand
and refine the set of heuristics to select an uncertainty estimation technique,
introducing tests for clinically-relevant scenarios such as generalization to
uncommon pathologies, changes in clinical protocol and simulations of corrupted
data. We furthermore differentiate these heuristics depending on the clinical
use-case. Second, we observe that ensembles and related techniques perform
poorly when it comes to detecting out-of-domain examples, a critical task which
is carried out more successfully by auto-encoders. These remarks are enriched
by considerations of the interplay of uncertainty estimation with class
imbalance, post-modeling calibration and other modeling procedures. Our
findings are supported by an array of experiments on toy and real-world data.
