  Since the development of self-supervised visual representation learning from
contrastive learning to masked image modeling (MIM), there is no significant
difference in essence, that is, how to design proper pretext tasks for vision
dictionary look-up. MIM recently dominates this line of research with
state-of-the-art performance on vision Transformers (ViTs), where the core is
to enhance the patch-level visual context capturing of the network via
denoising auto-encoding mechanism. Rather than tailoring image tokenizers with
extra training stages as in previous works, we unleash the great potential of
contrastive learning on denoising auto-encoding and introduce a pure MIM
method, ConMIM, to produce simple intra-image inter-patch contrastive
constraints as the sole learning objectives for masked patch prediction. We
further strengthen the denoising mechanism with asymmetric designs, including
image perturbations and model progress rates, to improve the network
pre-training. ConMIM-pretrained models with various scales achieve competitive
results on downstream image classification, semantic segmentation, object
detection, and instance segmentation tasks, e.g., on ImageNet-1K
classification, we achieve 83.9% top-1 accuracy with ViT-Small and 85.3% with
ViT-Base without extra data for pre-training.
