  We introduce a framework for learning robust visual representations that
generalize to new viewpoints, backgrounds, and scene contexts. Discriminative
models often learn naturally occurring spurious correlations, which cause them
to fail on images outside of the training distribution. In this paper, we show
that we can steer generative models to manufacture interventions on features
caused by confounding factors. Experiments, visualizations, and theoretical
results show this method learns robust representations more consistent with the
underlying causal relationships. Our approach improves performance on multiple
datasets demanding out-of-distribution generalization, and we demonstrate
state-of-the-art performance generalizing from ImageNet to ObjectNet dataset.
