  Neural architectures based on attention such as vision transformers are
revolutionizing image recognition. Their main benefit is that attention allows
reasoning about all parts of a scene jointly. In this paper, we show how the
global reasoning of (scaled) dot-product attention can be the source of a major
vulnerability when confronted with adversarial patch attacks. We provide a
theoretical understanding of this vulnerability and relate it to an adversary's
ability to misdirect the attention of all queries to a single key token under
the control of the adversarial patch. We propose novel adversarial objectives
for crafting adversarial patches which target this vulnerability explicitly. We
show the effectiveness of the proposed patch attacks on popular image
classification (ViTs and DeiTs) and object detection models (DETR). We find
that adversarial patches occupying 0.5% of the input can lead to robust
accuracies as low as 0% for ViT on ImageNet, and reduce the mAP of DETR on MS
COCO to less than 3%.
