  In many classification datasets, the task labels are spuriously correlated
with some input attributes. Classifiers trained on such datasets often rely on
these attributes for prediction, especially when the spurious correlation is
high, and thus fail to generalize whenever there is a shift in the attributes'
correlation at deployment. If we assume that the spurious attributes are known
a priori, several methods have been proposed to learn a classifier that is
invariant to the specified attributes. However, in real-world data, information
about spurious attributes is typically unavailable. Therefore, we propose a
method to automatically identify spurious attributes by estimating their causal
effect on the label and then use a regularization objective to mitigate the
classifier's reliance on them. Compared to a recent method for identifying
spurious attributes, we find that our method is more accurate in removing the
attribute from the learned model, especially when spurious correlation is high.
Specifically, across synthetic, semi-synthetic, and real-world datasets, our
method shows significant improvement in a metric used to quantify the
dependence of a classifier on spurious attributes ($\Delta$Prob), while
obtaining better or similar accuracy. In addition, our method mitigates the
reliance on spurious attributes even under noisy estimation of causal effects.
To explain the empirical robustness of our method, we create a simple linear
classification task with two sets of attributes: causal and spurious. We prove
that our method only requires that the ranking of estimated causal effects is
correct across attributes to select the correct classifier.
