  Vision Transformers (ViTs) achieve superior performance on various tasks
compared to convolutional neural networks (CNNs), but ViTs are also vulnerable
to adversarial attacks. Adversarial training is one of the most successful
methods to build robust CNN models. Thus, recent works explored new
methodologies for adversarial training of ViTs based on the differences between
ViTs and CNNs, such as better training strategies, preventing attention from
focusing on a single block, or discarding low-attention embeddings. However,
these methods still follow the design of traditional supervised adversarial
training, limiting the potential of adversarial training on ViTs. This paper
proposes a novel defense method, MIMIR, which aims to build a different
adversarial training methodology by utilizing Masked Image Modeling at
pre-training. We create an autoencoder that accepts adversarial examples as
input but takes the clean examples as the modeling target. Then, we create a
mutual information (MI) penalty following the idea of the Information
Bottleneck. Among the two information source inputs and corresponding
adversarial perturbation, the perturbation information is eliminated due to the
constraint of the modeling target. Next, we provide a theoretical analysis of
MIMIR using the bounds of the MI penalty. We also design two adaptive attacks
when the adversary is aware of the MIMIR defense and show that MIMIR still
performs well. The experimental results show that MIMIR improves (natural and
adversarial) accuracy on average by 4.19% on CIFAR-10 and 5.52% on ImageNet-1K,
compared to baselines. On Tiny-ImageNet, we obtained improved natural accuracy
of 2.99\% on average and comparable adversarial accuracy. Our code and trained
models are publicly available https://github.com/xiaoyunxxy/MIMIR.
