  Self-supervised contrastive learning is a powerful tool to learn visual
representation without labels. Prior work has primarily focused on evaluating
the recognition accuracy of various pre-training algorithms, but has overlooked
other behavioral aspects. In addition to accuracy, distributional robustness
plays a critical role in the reliability of machine learning models. We design
and conduct a series of robustness tests to quantify the behavioral differences
between contrastive learning and supervised learning to downstream or
pre-training data distribution changes. These tests leverage data corruptions
at multiple levels, ranging from pixel-level gamma distortion to patch-level
shuffling and to dataset-level distribution shift. Our tests unveil intriguing
robustness behaviors of contrastive and supervised learning. On the one hand,
under downstream corruptions, we generally observe that contrastive learning is
surprisingly more robust than supervised learning. On the other hand, under
pre-training corruptions, we find contrastive learning vulnerable to patch
shuffling and pixel intensity change, yet less sensitive to dataset-level
distribution change. We attempt to explain these results through the role of
data augmentation and feature space properties. Our insight has implications in
improving the downstream robustness of supervised learning.
