  Large pre-trained Vision-Language Models (VLMs) like CLIP, despite having
remarkable generalization ability, are highly vulnerable to adversarial
examples. This work studies the adversarial robustness of VLMs from the novel
perspective of the text prompt instead of the extensively studied model weights
(frozen in this work). We first show that the effectiveness of both adversarial
attack and defense are sensitive to the used text prompt. Inspired by this, we
propose a method to improve resilience to adversarial attacks by learning a
robust text prompt for VLMs. The proposed method, named Adversarial Prompt
Tuning (APT), is effective while being both computationally and data efficient.
Extensive experiments are conducted across 15 datasets and 4 data sparsity
schemes (from 1-shot to full training data settings) to show APT's superiority
over hand-engineered prompts and other state-of-the-art adaption methods. APT
demonstrated excellent abilities in terms of the in-distribution performance
and the generalization under input distribution shift and across datasets.
Surprisingly, by simply adding one learned word to the prompts, APT can
significantly boost the accuracy and robustness (epsilon=4/255) over the
hand-engineered prompts by +13% and +8.5% on average respectively. The
improvement further increases, in our most effective setting, to +26.4% for
accuracy and +16.7% for robustness. Code is available at
https://github.com/TreeLLi/APT.
