  Masked Image Modeling (MIM) methods, like Masked Autoencoders (MAE),
efficiently learn a rich representation of the input. However, for adapting to
downstream tasks, they require a sufficient amount of labeled data since their
rich features code not only objects but also less relevant image background. In
contrast, Instance Discrimination (ID) methods focus on objects. In this work,
we study how to combine the efficiency and scalability of MIM with the ability
of ID to perform downstream classification in the absence of large amounts of
labeled data. To this end, we introduce Masked Autoencoder Contrastive Tuning
(MAE-CT), a sequential approach that utilizes the implicit clustering of the
Nearest Neighbor Contrastive Learning (NNCLR) objective to induce abstraction
in the topmost layers of a pre-trained MAE. MAE-CT tunes the rich features such
that they form semantic clusters of objects without using any labels. Notably,
MAE-CT does not rely on hand-crafted augmentations and frequently achieves its
best performances while using only minimal augmentations (crop & flip).
Further, MAE-CT is compute efficient as it requires at most 10% overhead
compared to MAE re-training. Applied to large and huge Vision Transformer (ViT)
models, MAE-CT excels over previous self-supervised methods trained on ImageNet
in linear probing, k-NN and low-shot classification accuracy as well as in
unsupervised clustering accuracy. With ViT-H/16 MAE-CT achieves a new
state-of-the-art in linear probing of 82.2%.
