  Trust is a crucial factor affecting the adoption of machine learning (ML)
models. Qualitative studies have revealed that end-users, particularly in the
medical domain, need models that can express their uncertainty in
decision-making allowing users to know when to ignore the model's
recommendations. However, existing approaches for quantifying decision-making
uncertainty are not model-agnostic, or they rely on complex statistical
derivations that are not easily understood by laypersons or end-users, making
them less useful for explaining the model's decision-making process. This work
proposes a set of class-independent meta-heuristics that can characterize the
complexity of an instance in terms of factors are mutually relevant to both
human and ML decision-making. The measures are integrated into a meta-learning
framework that estimates the risk of misclassification. The proposed framework
outperformed predicted probabilities in identifying instances at risk of being
misclassified. The proposed measures and framework hold promise for improving
model development for more complex instances, as well as providing a new means
of model abstention and explanation.
