  Recent advancements in masked image modeling (MIM) have made it a prevailing
framework for self-supervised visual representation learning. The MIM
pretrained models, like most deep neural network methods, remain vulnerable to
adversarial attacks, limiting their practical application, and this issue has
received little research attention. In this paper, we investigate how this
powerful self-supervised learning paradigm can provide adversarial robustness
to downstream classifiers. During the exploration, we find that noisy image
modeling (NIM), a simple variant of MIM that adopts denoising as the pre-text
task, reconstructs noisy images surprisingly well despite severe corruption.
Motivated by this observation, we propose an adversarial defense method,
referred to as De^3, by exploiting the pretrained decoder for denoising.
Through De^3, NIM is able to enhance adversarial robustness beyond providing
pretrained features. Furthermore, we incorporate a simple modification,
sampling the noise scale hyperparameter from random distributions, and enable
the defense to achieve a better and tunable trade-off between accuracy and
robustness. Experimental results demonstrate that, in terms of adversarial
robustness, NIM is superior to MIM thanks to its effective denoising
capability. Moreover, the defense provided by NIM achieves performance on par
with adversarial training while offering the extra tunability advantage. Source
code and models are available at https://github.com/youzunzhi/NIM-AdvDef.
