  Uncertainty estimation is critical for cost-sensitive deep-learning
applications (i.e. disease diagnosis). It is very challenging partly due to the
inaccessibility of uncertainty groundtruth in most datasets. Previous works
proposed to estimate the uncertainty from softmax calibration, Monte Carlo
sampling, subjective logic and so on. However, these existing methods tend to
be over-confident about their predictions with unreasonably low overall
uncertainty, which originates from the imbalance between positive (correct
classifications) and negative (incorrect classifications) samples. For this
issue, we firstly propose the distributional imbalance to model the imbalance
in uncertainty estimation as two kinds of distribution biases, and secondly
propose Balanced True Class Probability (BTCP) framework, which learns an
uncertainty estimator with a novel Distributional Focal Loss (DFL) objective.
Finally, we evaluate the BTCP in terms of failure prediction and
out-of-distribution (OOD) detection on multiple datasets. The experimental
results show that BTCP outperforms other uncertainty estimation methods
especially in identifying incorrect classifications.
