  In this paper, we present a comparative analysis of various self-supervised
Vision Transformers (ViTs), focusing on their local representative power.
Inspired by large language models, we examine the abilities of ViTs to perform
various computer vision tasks with little to no fine-tuning. We design
evaluation framework to analyze the quality of local, i.e.\ patch-level,
representations in the context of few-shot semantic segmentation, instance
identification, object retrieval and tracking. We discover that contrastive
learning based methods like DINO produce more universal patch representations
that can be immediately applied for downstream tasks with no parameter tuning,
compared to masked image modeling. The embeddings learned using the latter
approach, e.g. in masked autoencoders, have high variance features that harm
distance-based algorithms, such as k-NN, and do not contain useful information
for most downstream tasks. Furthermore, we demonstrate that removing these
high-variance features enhances k-NN for MAE, as well as for its recent
extension Scale-MAE. Finally, we find an object instance retrieval setting
where DINOv2, a model pretrained on two orders of magnitude more data, falls
short of its less compute intensive counterpart DINO.
