  Self-supervised learning has emerged as a strategy to reduce the reliance on
costly supervised signal by pretraining representations only using unlabeled
data. These methods combine heuristic proxy classification tasks with data
augmentations and have achieved significant success, but our theoretical
understanding of this success remains limited. In this paper we analyze
self-supervised representation learning using a causal framework. We show how
data augmentations can be more effectively utilized through explicit invariance
constraints on the proxy classifiers employed during pretraining. Based on
this, we propose a novel self-supervised objective, Representation Learning via
Invariant Causal Mechanisms (ReLIC), that enforces invariant prediction of
proxy targets across augmentations through an invariance regularizer which
yields improved generalization guarantees. Further, using causality we
generalize contrastive learning, a particular kind of self-supervised method,
and provide an alternative theoretical explanation for the success of these
methods. Empirically, ReLIC significantly outperforms competing methods in
terms of robustness and out-of-distribution generalization on ImageNet, while
also significantly outperforming these methods on Atari achieving above
human-level performance on $51$ out of $57$ games.
