  Existing works show that although modern neural networks achieve remarkable
generalization performance on the in-distribution (ID) dataset, the accuracy
drops significantly on the out-of-distribution (OOD) datasets
\cite{recht2018cifar, recht2019imagenet}. To understand why a variety of models
consistently make more mistakes in the OOD datasets, we propose a new metric to
quantify the difficulty of the test images (either ID or OOD) that depends on
the interaction of the training dataset and the model. In particular, we
introduce \textit{confusion score} as a label-free measure of image difficulty
which quantifies the amount of disagreement on a given test image based on the
class conditional probabilities estimated by an ensemble of trained models.
Using the confusion score, we investigate CIFAR-10 and its OOD derivatives.
Next, by partitioning test and OOD datasets via their confusion scores, we
predict the relationship between ID and OOD accuracies for various
architectures. This allows us to obtain an estimator of the OOD accuracy of a
given model only using ID test labels. Our observations indicate that the
biggest contribution to the accuracy drop comes from images with high confusion
scores. Upon further inspection, we report on the nature of the misclassified
images grouped by their confusion scores: \textit{(i)} images with high
confusion scores contain \textit{weak spurious correlations} that appear in
multiple classes in the training data and lack clear \textit{class-specific
features}, and \textit{(ii)} images with low confusion scores exhibit spurious
correlations that belong to another class, namely \textit{class-specific
spurious correlations}.
