  In medical diagnostics, accurate uncertainty estimation for neural-based
models is essential for complementing second-opinion systems. Despite neural
network ensembles' proficiency in this problem, a gap persists between actual
uncertainties and predicted estimates. A major difficulty here is the lack of
labels on the hardness of examples: a typical dataset includes only ground
truth target labels, making the uncertainty estimation problem almost
unsupervised. Our novel approach narrows this gap by integrating expert
assessments of case complexity into the neural network's learning process,
utilizing both definitive target labels and supplementary complexity ratings.
We validate our methodology for blood typing, leveraging a new dataset
"BloodyWell" unique in augmenting labeled reaction images with complexity
scores from six medical specialists. Experiments demonstrate enhancement of our
approach in uncertainty prediction, achieving a 2.5-fold improvement with
expert labels and a 35% increase in performance with estimates of neural-based
expert consensus.
