  Spurious patterns refer to a mathematical association between two or more
variables in a dataset that are not causally related. However, this notion of
spuriousness, which is usually introduced due to sampling biases in the
dataset, has classically lacked a formal definition. To address this gap, this
work presents the first information-theoretic formalization of spuriousness in
a dataset (given a split of spurious and core features) using a mathematical
framework called Partial Information Decomposition (PID). Specifically, we
disentangle the joint information content that the spurious and core features
share about another target variable (e.g., the prediction label) into distinct
components, namely unique, redundant, and synergistic information. We propose
the use of unique information, with roots in Blackwell Sufficiency, as a novel
metric to formally quantify dataset spuriousness and derive its desirable
properties. We empirically demonstrate how higher unique information in the
spurious features in a dataset could lead a model into choosing the spurious
features over the core features for inference, often having low
worst-group-accuracy. We also propose a novel autoencoder-based estimator for
computing unique information that is able to handle high-dimensional image
data. Finally, we also show how this unique information in the spurious feature
is reduced across several dataset-based spurious-pattern-mitigation techniques
such as data reweighting and varying levels of background mixing, demonstrating
a novel tradeoff between unique information (spuriousness) and
worst-group-accuracy.
