  Uncertainty quantification in automated image analysis is highly desired in
many applications. Typically, machine learning models in classification or
segmentation are only developed to provide binary answers; however, quantifying
the uncertainty of the models can play a critical role for example in active
learning or machine human interaction. Uncertainty quantification is especially
difficult when using deep learning-based models, which are the state-of-the-art
in many imaging applications. The current uncertainty quantification approaches
do not scale well in high-dimensional real-world problems. Scalable solutions
often rely on classical techniques, such as dropout, during inference or
training ensembles of identical models with different random seeds to obtain a
posterior distribution. In this paper, we show that these approaches fail to
approximate the classification probability. On the contrary, we propose a
scalable and intuitive framework to calibrate ensembles of deep learning models
to produce uncertainty quantification measurements that approximate the
classification probability. On unseen test data, we demonstrate improved
calibration, sensitivity (in two out of three cases) and precision when being
compared with the standard approaches. We further motivate the usage of our
method in active learning, creating pseudo-labels to learn from unlabeled
images and human-machine collaboration.
