  Recently, Masked Image Modeling (MIM) achieves great success in
self-supervised visual recognition. However, as a reconstruction-based
framework, it is still an open question to understand how MIM works, since MIM
appears very different from previous well-studied siamese approaches such as
contrastive learning. In this paper, we propose a new viewpoint: MIM implicitly
learns occlusion-invariant features, which is analogous to other siamese
methods while the latter learns other invariance. By relaxing MIM formulation
into an equivalent siamese form, MIM methods can be interpreted in a unified
framework with conventional methods, among which only a) data transformations,
i.e. what invariance to learn, and b) similarity measurements are different.
Furthermore, taking MAE (He et al.) as a representative example of MIM, we
empirically find the success of MIM models relates a little to the choice of
similarity functions, but the learned occlusion invariant feature introduced by
masked image -- it turns out to be a favored initialization for vision
transformers, even though the learned feature could be less semantic. We hope
our findings could inspire researchers to develop more powerful self-supervised
methods in computer vision community.
