  Self-Supervised Learning (SSL) is a paradigm that leverages unlabeled data
for model training. Empirical studies show that SSL can achieve promising
performance in distribution shift scenarios, where the downstream and training
distributions differ. However, the theoretical understanding of its
transferability remains limited. In this paper, we develop a theoretical
framework to analyze the transferability of self-supervised contrastive
learning, by investigating the impact of data augmentation on it. Our results
reveal that the downstream performance of contrastive learning depends largely
on the choice of data augmentation. Moreover, we show that contrastive learning
fails to learn domain-invariant features, which limits its transferability.
Based on these theoretical insights, we propose a novel method called
Augmentation-robust Contrastive Learning (ArCL), which guarantees to learn
domain-invariant features and can be easily integrated with existing
contrastive learning algorithms. We conduct experiments on several datasets and
show that ArCL significantly improves the transferability of contrastive
learning.
