  Masked image modeling (MIM) learns visual representation by masking and
reconstructing image patches. Applying the reconstruction supervision on the
CLIP representation has been proven effective for MIM. However, it is still
under-explored how CLIP supervision in MIM influences performance. To
investigate strategies for refining the CLIP-targeted MIM, we study two
critical elements in MIM, i.e., the supervision position and the mask ratio,
and reveal two interesting perspectives, relying on our developed simple
pipeline, context autodecoder with CLIP target (CAE v2). Firstly, we observe
that the supervision on visible patches achieves remarkable performance, even
better than that on masked patches, where the latter is the standard format in
the existing MIM methods. Secondly, the optimal mask ratio positively
correlates to the model size. That is to say, the smaller the model, the lower
the mask ratio needs to be. Driven by these two discoveries, our simple and
concise approach CAE v2 achieves superior performance on a series of downstream
tasks. For example, a vanilla ViT-Large model achieves 81.7% and 86.7% top-1
accuracy on linear probing and fine-tuning on ImageNet-1K, and 55.9% mIoU on
semantic segmentation on ADE20K with the pre-training for 300 epochs. We hope
our findings can be helpful guidelines for the pre-training in the MIM area,
especially for the small-scale models.
