  Deep neural networks may be susceptible to learning spurious correlations
that hold on average but not in atypical test samples. As with the recent
emergence of vision transformer (ViT) models, it remains underexplored how
spurious correlations are manifested in such architectures. In this paper, we
systematically investigate the robustness of vision transformers to spurious
correlations on three challenging benchmark datasets and compare their
performance with popular CNNs. Our study reveals that when pre-trained on a
sufficiently large dataset, ViT models are more robust to spurious correlations
than CNNs. Key to their success is the ability to generalize better from the
examples where spurious correlations do not hold. Further, we perform extensive
ablations and experiments to understand the role of the self-attention
mechanism in providing robustness under spuriously correlated environments. We
hope that our work will inspire future research on further understanding the
robustness of ViT models.
