  Recently, NLP models have achieved remarkable progress across a variety of
tasks; however, they have also been criticized for being not robust. Many
robustness problems can be attributed to models exploiting spurious
correlations, or shortcuts between the training data and the task labels. Most
existing work identifies a limited set of task-specific shortcuts via human
priors or error analyses, which requires extensive expertise and efforts. In
this paper, we aim to automatically identify such spurious correlations in NLP
models at scale. We first leverage existing interpretability methods to extract
tokens that significantly affect model's decision process from the input text.
We then distinguish "genuine" tokens and "spurious" tokens by analyzing model
predictions across multiple corpora and further verify them through
knowledge-aware perturbations. We show that our proposed method can effectively
and efficiently identify a scalable set of "shortcuts", and mitigating these
leads to more robust models in multiple applications.
