  Despite alarm over the reliance of machine learning systems on so-called
spurious patterns, the term lacks coherent meaning in standard statistical
frameworks. However, the language of causality offers clarity: spurious
associations are due to confounding (e.g., a common cause), but not direct or
indirect causal effects. In this paper, we focus on natural language
processing, introducing methods and resources for training models less
sensitive to spurious patterns. Given documents and their initial labels, we
task humans with revising each document so that it (i) accords with a
counterfactual target label; (ii) retains internal coherence; and (iii) avoids
unnecessary changes. Interestingly, on sentiment analysis and natural language
inference tasks, classifiers trained on original data fail on their
counterfactually-revised counterparts and vice versa. Classifiers trained on
combined datasets perform remarkably well, just shy of those specialized to
either domain. While classifiers trained on either original or manipulated data
alone are sensitive to spurious features (e.g., mentions of genre), models
trained on the combined data are less sensitive to this signal. Both datasets
are publicly available.
