  We study the problem of out-of-distribution (o.o.d.) generalization where
spurious correlations of attributes vary across training and test domains. This
is known as the problem of correlation shift and has posed concerns on the
reliability of machine learning. In this work, we introduce the concepts of
direct and indirect effects from causal inference to the domain generalization
problem. We argue that models that learn direct effects minimize the worst-case
risk across correlation-shifted domains. To eliminate the indirect effects, our
algorithm consists of two stages: in the first stage, we learn an
indirect-effect representation by minimizing the prediction error of domain
labels using the representation and the class labels; in the second stage, we
remove the indirect effects learned in the first stage by matching each data
with another data of similar indirect-effect representation but of different
class labels in the training and validation phase. Our approach is shown to be
compatible with existing methods and improve the generalization performance of
them on correlation-shifted datasets. Experiments on 5 correlation-shifted
datasets and the DomainBed benchmark verify the effectiveness of our approach.
