  We introduce MIM (Masked Image Modeling)-Refiner, a contrastive learning
boost for pre-trained MIM models. MIM-Refiner is motivated by the insight that
strong representations within MIM models generally reside in intermediate
layers. Accordingly, MIM-Refiner leverages multiple contrastive heads that are
connected to different intermediate layers. In each head, a modified nearest
neighbor objective constructs semantic clusters that capture semantic
information which improves performance on downstream tasks, including
off-the-shelf and fine-tuning settings.
  The refinement process is short and simple - yet highly effective. Within a
few epochs, we refine the features of MIM models from subpar to
state-of-the-art, off-the-shelf features. Refining a ViT-H, pre-trained with
data2vec 2.0 on ImageNet-1K, sets a new state-of-the-art in linear probing
(84.7%) and low-shot classification among models that are pre-trained on
ImageNet-1K. At ImageNet-1K 1-shot classification, MIM-Refiner advances the
state-of-the-art to 64.2%, outperforming larger models that were trained on up
to 2000 times more data such as DINOv2-g, OpenCLIP-G and MAWS-6.5B.
