  Recent studies have revealed that vision transformers (ViTs) face similar
security risks from adversarial attacks as deep convolutional neural networks
(CNNs). However, directly applying attack methodology on CNNs to ViTs has been
demonstrated to be ineffective since the ViTs typically work on patch-wise
encoding. This article explores the vulnerability of ViTs against adversarial
attacks under a black-box scenario, and proposes a novel query-efficient
hard-label adversarial attack method called AdvViT. Specifically, considering
that ViTs are highly sensitive to patch modification, we propose to optimize
the adversarial perturbation on the individual patches. To reduce the dimension
of perturbation search space, we modify only a handful of low-frequency
components of each patch. Moreover, we design a weight mask matrix for all
patches to further optimize the perturbation on different regions of a whole
image. We test six mainstream ViT backbones on the ImageNet-1k dataset.
Experimental results show that compared with the state-of-the-art attacks on
CNNs, our AdvViT achieves much lower $L_2$-norm distortion under the same query
budget, sufficiently validating the vulnerability of ViTs against adversarial
attacks.
