  The latest generation of transformer-based vision models has proven to be
superior to Convolutional Neural Network (CNN)-based models across several
vision tasks, largely attributed to their remarkable prowess in relation
modeling. Deformable vision transformers significantly reduce the quadratic
complexity of attention modeling by using sparse attention structures, enabling
them to incorporate features across different scales and be used in large-scale
applications, such as multi-view vision systems. Recent work has demonstrated
adversarial attacks against conventional vision transformers; we show that
these attacks do not transfer to deformable transformers due to their sparse
attention structure. Specifically, attention in deformable transformers is
modeled using pointers to the most relevant other tokens. In this work, we
contribute for the first time adversarial attacks that manipulate the attention
of deformable transformers, redirecting it to focus on irrelevant parts of the
image. We also develop new collaborative attacks where a source patch
manipulates attention to point to a target patch, which contains the
adversarial noise to fool the model. In our experiments, we observe that
altering less than 1% of the patched area in the input field results in a
complete drop to 0% AP in single-view object detection using MS COCO and a 0%
MODA in multi-view object detection using Wildtrack.
