  Self-supervised learning (SSL) has emerged as a desirable paradigm in
computer vision due to the inability of supervised models to learn
representations that can generalize in domains with limited labels. The recent
popularity of SSL has led to the development of several models that make use of
diverse training strategies, architectures, and data augmentation policies with
no existing unified framework to study or assess their effectiveness in
transfer learning. We propose a data-driven geometric strategy to analyze
different SSL models using local neighborhoods in the feature space induced by
each. Unlike existing approaches that consider mathematical approximations of
the parameters, individual components, or optimization landscape, our work aims
to explore the geometric properties of the representation manifolds learned by
SSL models. Our proposed manifold graph metrics (MGMs) provide insights into
the geometric similarities and differences between available SSL models, their
invariances with respect to specific augmentations, and their performances on
transfer learning tasks. Our key findings are two fold: (i) contrary to popular
belief, the geometry of SSL models is not tied to its training paradigm
(contrastive, non-contrastive, and cluster-based); (ii) we can predict the
transfer learning capability for a specific model based on the geometric
properties of its semantic and augmentation manifolds.
