  Self-supervised learning (SSL) has emerged as a powerful technique for
learning rich representations from unlabeled data. The data representations are
able to capture many underlying attributes of data, and be useful in downstream
prediction tasks. In real-world settings, spurious correlations between some
attributes (e.g. race, gender and age) and labels for downstream tasks often
exist, e.g. cancer is usually more prevalent among elderly patients. In this
paper, we investigate SSL in the presence of spurious correlations and show
that the SSL training loss can be minimized by capturing only a subset of the
conspicuous features relevant to those sensitive attributes, despite the
presence of other important predictive features for the downstream tasks. To
address this issue, we investigate the learning dynamics of SSL and observe
that the learning is slower for samples that conflict with such correlations
(e.g. elder patients without cancer). Motivated by these findings, we propose a
learning-speed aware SSL (LA-SSL) approach, in which we sample each training
data with a probability that is inversely related to its learning speed. We
evaluate LA-SSL on three datasets that exhibit spurious correlations between
different attributes, demonstrating that it improves the robustness of
pretrained representations on downstream classification tasks.
