  The problem of spurious correlations (SCs) arises when a classifier relies on
non-predictive features that happen to be correlated with the labels in the
training data. For example, a classifier may misclassify dog breeds based on
the background of dog images. This happens when the backgrounds are correlated
with other breeds in the training data, leading to misclassifications during
test time. Previous SC benchmark datasets suffer from varying issues, e.g.,
over-saturation or only containing one-to-one (O2O) SCs, but no many-to-many
(M2M) SCs arising between groups of spurious attributes and classes. In this
paper, we present \benchmark-\{O2O, M2M\}-\{Easy, Medium, Hard\}, an image
classification benchmark suite containing spurious correlations between classes
and backgrounds. To create this dataset, we employ a text-to-image model to
generate photo-realistic images and an image captioning model to filter out
unsuitable ones. The resulting dataset is of high quality and contains
approximately 152k images. Our experimental results demonstrate that
state-of-the-art group robustness methods struggle with \benchmark, most
notably on the Hard-splits with none of them getting over $70\%$ accuracy on
the hardest split using a ResNet50 pretrained on ImageNet. By examining model
misclassifications, we detect reliances on spurious backgrounds, demonstrating
that our dataset provides a significant challenge.
