  Self-supervised learning (SSL) is a machine learning approach where the data
itself provides supervision, eliminating the need for external labels. The
model is forced to learn about the data structure or context by solving a
pretext task. With SSL, models can learn from abundant and cheap unlabeled
data, significantly reducing the cost of training models where labels are
expensive or inaccessible. In Computer Vision, SSL is widely used as
pre-training followed by a downstream task, such as supervised transfer,
few-shot learning on smaller labeled data sets, and/or unsupervised clustering.
Unfortunately, it is infeasible to evaluate SSL methods on all possible
downstream tasks and objectively measure the quality of the learned
representation. Instead, SSL methods are evaluated using in-domain evaluation
protocols, such as fine-tuning, linear probing, and k-nearest neighbors (kNN).
However, it is not well understood how well these evaluation protocols estimate
the representation quality of a pre-trained model for different downstream
tasks under different conditions, such as dataset, metric, and model
architecture. We study how classification-based evaluation protocols for SSL
correlate and how well they predict downstream performance on different dataset
types. Our study includes eleven common image datasets and 26 models that were
pre-trained with different SSL methods or have different model backbones. We
find that in-domain linear/kNN probing protocols are, on average, the best
general predictors for out-of-domain performance. We further investigate the
importance of batch normalization and evaluate how robust correlations are for
different kinds of dataset domain shifts. We challenge assumptions about the
relationship between discriminative and generative self-supervised methods,
finding that most of their performance differences can be explained by changes
to model backbones.
