  Although much of the success of Deep Learning builds on learning good
representations, a rigorous method to evaluate their quality is lacking. In
this paper, we treat the evaluation of representations as a model selection
problem and propose to use the Minimum Description Length (MDL) principle to
devise an evaluation metric. Contrary to the established practice of limiting
the capacity of the readout model, we design a hybrid discrete and
continuous-valued model space for the readout models and employ a switching
strategy to combine their predictions. The MDL score takes model complexity, as
well as data efficiency into account. As a result, the most appropriate model
for the specific task and representation will be chosen, making it a unified
measure for comparison. The proposed metric can be efficiently computed with an
online method and we present results for pre-trained vision encoders of various
architectures (ResNet and ViT) and objective functions (supervised and
self-supervised) on a range of downstream tasks. We compare our methods with
accuracy-based approaches and show that the latter are inconsistent when
multiple readout models are used. Finally, we discuss important properties
revealed by our evaluations such as model scaling, preferred readout model, and
data efficiency.
