  Masked image modeling (MIM) has gained significant traction for its
remarkable prowess in representation learning. As an alternative to the
traditional approach, the reconstruction from corrupted images has recently
emerged as a promising pretext task. However, the regular corrupted images are
generated using generic generators, often lacking relevance to the specific
reconstruction task involved in pre-training. Hence, reconstruction from
regular corrupted images cannot ensure the difficulty of the pretext task,
potentially leading to a performance decline. Moreover, generating corrupted
images might introduce an extra generator, resulting in a notable computational
burden. To address these issues, we propose to incorporate adversarial examples
into masked image modeling, as the new reconstruction targets. Adversarial
examples, generated online using only the trained models, can directly aim to
disrupt tasks associated with pre-training. Therefore, the incorporation not
only elevates the level of challenge in reconstruction but also enhances
efficiency, contributing to the acquisition of superior representations by the
model. In particular, we introduce a novel auxiliary pretext task that
reconstructs the adversarial examples corresponding to the original images. We
also devise an innovative adversarial attack to craft more suitable adversarial
examples for MIM pre-training. It is noted that our method is not restricted to
specific model architectures and MIM strategies, rendering it an adaptable
plug-in capable of enhancing all MIM methods. Experimental findings
substantiate the remarkable capability of our approach in amplifying the
generalization and robustness of existing MIM methods. Notably, our method
surpasses the performance of baselines on various tasks, including ImageNet,
its variants, and other downstream tasks.
