  Understanding the mechanisms behind Vision Transformer (ViT), particularly
its vulnerability to adversarial perturba tions, is crucial for addressing
challenges in its real-world applications. Existing ViT adversarial attackers
rely on la bels to calculate the gradient for perturbation, and exhibit low
transferability to other structures and tasks. In this paper, we present a
label-free white-box attack approach for ViT-based models that exhibits strong
transferability to various black box models, including most ViT variants, CNNs,
and MLPs, even for models developed for other modalities. Our inspira tion
comes from the feature collapse phenomenon in ViTs, where the critical
attention mechanism overly depends on the low-frequency component of features,
causing the features in middle-to-end layers to become increasingly similar and
eventually collapse. We propose the feature diversity attacker to naturally
accelerate this process and achieve remarkable performance and transferability.
