  Recent advances in self-supervised learning and neural network scaling have
enabled the creation of large models, known as foundation models, which can be
easily adapted to a wide range of downstream tasks. The current paradigm for
comparing foundation models involves evaluating them with aggregate metrics on
various benchmark datasets. This method of model comparison is heavily
dependent on the chosen evaluation metric, which makes it unsuitable for
situations where the ideal metric is either not obvious or unavailable. In this
work, we present a methodology for directly comparing the embedding space
geometry of foundation models, which facilitates model comparison without the
need for an explicit evaluation metric. Our methodology is grounded in random
graph theory and enables valid hypothesis testing of embedding similarity on a
per-datum basis. Further, we demonstrate how our methodology can be extended to
facilitate population level model comparison. In particular, we show how our
framework can induce a manifold of models equipped with a distance function
that correlates strongly with several downstream metrics. We remark on the
utility of this population level model comparison as a first step towards a
taxonomic science of foundation models.
