  Benchmark performance of deep learning classifiers alone is not a reliable
predictor for the performance of a deployed model. In particular, if the image
classifier has picked up spurious features in the training data, its
predictions can fail in unexpected ways. In this paper, we develop a framework
that allows us to systematically identify spurious features in large datasets
like ImageNet. It is based on our neural PCA components and their
visualization. Previous work on spurious features often operates in toy
settings or requires costly pixel-wise annotations. In contrast, we work with
ImageNet and validate our results by showing that presence of the harmful
spurious feature of a class alone is sufficient to trigger the prediction of
that class. We introduce the novel dataset "Spurious ImageNet" which allows to
measure the reliance of any ImageNet classifier on harmful spurious features.
Moreover, we introduce SpuFix as a simple mitigation method to reduce the
dependence of any ImageNet classifier on previously identified harmful spurious
features without requiring additional labels or retraining of the model. We
provide code and data at https://github.com/YanNeu/spurious_imagenet .
