  Vision Transformers (ViT) have recently demonstrated exemplary performance on
a variety of vision tasks and are being used as an alternative to CNNs. Their
design is based on a self-attention mechanism that processes images as a
sequence of patches, which is quite different compared to CNNs. Hence it is
interesting to study if ViTs are vulnerable to backdoor attacks. Backdoor
attacks happen when an attacker poisons a small part of the training data for
malicious purposes. The model performance is good on clean test images, but the
attacker can manipulate the decision of the model by showing the trigger at
test time. To the best of our knowledge, we are the first to show that ViTs are
vulnerable to backdoor attacks. We also find an intriguing difference between
ViTs and CNNs - interpretation algorithms effectively highlight the trigger on
test images for ViTs but not for CNNs. Based on this observation, we propose a
test-time image blocking defense for ViTs which reduces the attack success rate
by a large margin. Code is available here:
https://github.com/UCDvision/backdoor_transformer.git
