  Misclassification detection is an important problem in machine learning, as
it allows for the identification of instances where the model's predictions are
unreliable. However, conventional uncertainty measures such as Shannon entropy
do not provide an effective way to infer the real uncertainty associated with
the model's predictions. In this paper, we introduce a novel data-driven
measure of uncertainty relative to an observer for misclassification detection.
By learning patterns in the distribution of soft-predictions, our uncertainty
measure can identify misclassified samples based on the predicted class
probabilities. Interestingly, according to the proposed measure,
soft-predictions corresponding to misclassified instances can carry a large
amount of uncertainty, even though they may have low Shannon entropy. We
demonstrate empirical improvements over multiple image classification tasks,
outperforming state-of-the-art misclassification detection methods.
