  Many NLP datasets have been found to contain shortcuts: simple decision rules
that achieve surprisingly high accuracy. However, it is difficult to discover
shortcuts automatically. Prior work on automatic shortcut detection has focused
on enumerating features like unigrams or bigrams, which can find only low-level
shortcuts, or relied on post-hoc model interpretability methods like saliency
maps, which reveal qualitative patterns without a clear statistical
interpretation. In this work, we propose to use probabilistic grammars to
characterize and discover shortcuts in NLP datasets. Specifically, we use a
context-free grammar to model patterns in sentence classification datasets and
use a synchronous context-free grammar to model datasets involving sentence
pairs. The resulting grammars reveal interesting shortcut features in a number
of datasets, including both simple and high-level features, and automatically
identify groups of test examples on which conventional classifiers fail.
Finally, we show that the features we discover can be used to generate
diagnostic contrast examples and incorporated into standard robust optimization
methods to improve worst-group accuracy.
