  Vision-Language Models (VLMs) are a significant technique for Artificial
General Intelligence (AGI). With the fast growth of AGI, the security problem
become one of the most important challenges for VLMs. In this paper, through
extensive experiments, we demonstrate the vulnerability of the conventional
adaptation methods for VLMs, which may bring significant security risks. In
addition, as the size of the VLMs increases, performing conventional
adversarial adaptation techniques on VLMs results in high computational costs.
To solve these problems, we propose a parameter-efficient
\underline{Adv}ersarial adaptation method named \underline{AdvLoRA} by
\underline{Lo}w-\underline{R}ank \underline{A}daptation. At first, we
investigate and reveal the intrinsic low-rank property during the adversarial
adaptation for VLMs. Different from LoRA, we improve the efficiency and
robustness of adversarial adaptation by designing a novel reparameterizing
method based on parameter clustering and parameter alignment. In addition, an
adaptive parameter update strategy is proposed to further improve the
robustness. By these settings, our proposed AdvLoRA alleviates the model
security and high resource waste problems. Extensive experiments demonstrate
the effectiveness and efficiency of the AdvLoRA.
