  Estimating machine learning performance 'in the wild' is both an important
and unsolved problem. In this paper, we seek to examine, understand, and
predict the pointwise competence of classification models. Our contributions
are twofold: First, we establish a statistically rigorous definition of
competence that generalizes the common notion of classifier confidence; second,
we present the ALICE (Accurate Layerwise Interpretable Competence Estimation)
Score, a pointwise competence estimator for any classifier. By considering
distributional, data, and model uncertainty, ALICE empirically shows accurate
competence estimation in common failure situations such as class-imbalanced
datasets, out-of-distribution datasets, and poorly trained models. Our
contributions allow us to accurately predict the competence of any
classification model given any input and error function. We compare our score
with state-of-the-art confidence estimators such as model confidence and Trust
Score, and show significant improvements in competence prediction over these
methods on datasets such as DIGITS, CIFAR10, and CIFAR100.
