  Due to spurious correlations, machine learning systems often fail to
generalize to environments whose distributions differ from the ones used at
training time. Prior work addressing this, either explicitly or implicitly,
attempted to find a data representation that has an invariant relationship with
the target. This is done by leveraging a diverse set of training environments
to reduce the effect of spurious features and build an invariant predictor.
However, these methods have generalization guarantees only when both data
representation and classifiers come from a linear model class. We propose
invariant Causal Representation Learning (iCaRL), an approach that enables
out-of-distribution (OOD) generalization in the nonlinear setting (i.e.,
nonlinear representations and nonlinear classifiers). It builds upon a
practical and general assumption: the prior over the data representation (i.e.,
a set of latent variables encoding the data) given the target and the
environment belongs to general exponential family distributions. Based on this,
we show that it is possible to identify the data representation up to simple
transformations. We also prove that all direct causes of the target can be
fully discovered, which further enables us to obtain generalization guarantees
in the nonlinear setting. Extensive experiments on both synthetic and
real-world datasets show that our approach outperforms a variety of baseline
methods. Finally, in the discussion, we further explore the aforementioned
assumption and propose a more general hypothesis, called the Agnostic
Hypothesis: there exist a set of hidden causal factors affecting both inputs
and outcomes. The Agnostic Hypothesis can provide a unifying view of machine
learning. More importantly, it can inspire a new direction to explore a general
theory for identifying hidden causal factors, which is key to enabling the OOD
generalization guarantees.
