  Spurious correlations are a threat to the trustworthiness of natural language
processing systems, motivating research into methods for identifying and
eliminating them. However, addressing the problem of spurious correlations
requires more clarity on what they are and how they arise in language data.
Gardner et al (2021) argue that the compositional nature of language implies
that \emph{all} correlations between labels and individual "input features" are
spurious. This paper analyzes this proposal in the context of a toy example,
demonstrating three distinct conditions that can give rise to feature-label
correlations in a simple PCFG. Linking the toy example to a structured causal
model shows that (1) feature-label correlations can arise even when the label
is invariant to interventions on the feature, and (2) feature-label
correlations may be absent even when the label is sensitive to interventions on
the feature. Because input features will be individually correlated with labels
in all but very rare circumstances, domain knowledge must be applied to
identify spurious correlations that pose genuine robustness threats.
