  Vision Transformers (ViTs) have demonstrated the state-of-the-art performance
in various vision-related tasks. The success of ViTs motivates adversaries to
perform backdoor attacks on ViTs. Although the vulnerability of traditional
CNNs to backdoor attacks is well-known, backdoor attacks on ViTs are
seldom-studied. Compared to CNNs capturing pixel-wise local features by
convolutions, ViTs extract global context information through patches and
attentions. Na\"ively transplanting CNN-specific backdoor attacks to ViTs
yields only a low clean data accuracy and a low attack success rate. In this
paper, we propose a stealth and practical ViT-specific backdoor attack
$TrojViT$. Rather than an area-wise trigger used by CNN-specific backdoor
attacks, TrojViT generates a patch-wise trigger designed to build a Trojan
composed of some vulnerable bits on the parameters of a ViT stored in DRAM
memory through patch salience ranking and attention-target loss. TrojViT
further uses minimum-tuned parameter update to reduce the bit number of the
Trojan. Once the attacker inserts the Trojan into the ViT model by flipping the
vulnerable bits, the ViT model still produces normal inference accuracy with
benign inputs. But when the attacker embeds a trigger into an input, the ViT
model is forced to classify the input to a predefined target class. We show
that flipping only few vulnerable bits identified by TrojViT on a ViT model
using the well-known RowHammer can transform the model into a backdoored one.
We perform extensive experiments of multiple datasets on various ViT models.
TrojViT can classify $99.64\%$ of test images to a target class by flipping
$345$ bits on a ViT for ImageNet.Our codes are available at
https://github.com/mxzheng/TrojViT
