  Recently, self-supervised metric learning has raised attention for the
potential to learn a generic distance function. It overcomes the limitations of
conventional supervised one, e.g., scalability and label biases. Despite
progress in this domain, current benchmarks, incorporating a narrow scope of
classes, stop the nuanced evaluation of semantic representations. To bridge
this gap, we introduce a large-scale benchmark with diversity and granularity
of classes, Statistical Metric Learning Benchmark (SMLB) built upon
ImageNet-21K and WordNet. SMLB is designed to rigorously evaluate the
discriminative discernment and generalizability across more than 14M images,
20K classes, and 16K taxonomic nodes. Alongside, we propose novel evaluation
metrics -- `overlap' for separability and `aSTD' for consistency -- to measure
distance statistical information, which are efficient and robust to the change
of class number. Our benchmark offers a novel perspective of evaluating the
quality of representations beyond accuracy. Our findings reveal the limitations
of supervised learning and the class bias inherent in SSL models, offering
insights into potential areas for future model enhancement.
