  Deep learning vision systems are widely deployed across applications where
reliability is critical. However, even today's best models can fail to
recognize an object when its pose, lighting, or background varies. While
existing benchmarks surface examples challenging for models, they do not
explain why such mistakes arise. To address this need, we introduce ImageNet-X,
a set of sixteen human annotations of factors such as pose, background, or
lighting the entire ImageNet-1k validation set as well as a random subset of
12k training images. Equipped with ImageNet-X, we investigate 2,200 current
recognition models and study the types of mistakes as a function of model's (1)
architecture, e.g. transformer vs. convolutional, (2) learning paradigm, e.g.
supervised vs. self-supervised, and (3) training procedures, e.g., data
augmentation. Regardless of these choices, we find models have consistent
failure modes across ImageNet-X categories. We also find that while data
augmentation can improve robustness to certain factors, they induce spill-over
effects to other factors. For example, strong random cropping hurts robustness
on smaller objects. Together, these insights suggest to advance the robustness
of modern vision models, future research should focus on collecting additional
data and understanding data augmentation schemes. Along with these insights, we
release a toolkit based on ImageNet-X to spur further study into the mistakes
image recognition systems make.
