  Masked image modeling, an emerging self-supervised pre-training method, has
shown impressive success across numerous downstream vision tasks with Vision
transformers. Its underlying idea is simple: a portion of the input image is
masked out and then reconstructed via a pre-text task. However, the working
principle behind MIM is not well explained, and previous studies insist that
MIM primarily works for the Transformer family but is incompatible with CNNs.
In this work, we observe that MIM essentially teaches the model to learn better
middle-order interactions among patches for more generalized feature
extraction. We then propose an Architecture-Agnostic Masked Image Modeling
framework (A$^2$MIM), which is compatible with both Transformers and CNNs in a
unified way. Extensive experiments on popular benchmarks show that A$^2$MIM
learns better representations without explicit design and endows the backbone
model with the stronger capability to transfer to various downstream tasks.
