  With Vision-Language Pre-training (VLP) models demonstrating powerful
multimodal interaction capabilities, the application scenarios of neural
networks are no longer confined to unimodal domains but have expanded to more
complex multimodal V+L downstream tasks. The security vulnerabilities of
unimodal models have been extensively examined, whereas those of VLP models
remain challenging. We note that in CV models, the understanding of images
comes from annotated information, while VLP models are designed to learn image
representations directly from raw text. Motivated by this discrepancy, we
developed the Feature Guidance Attack (FGA), a novel method that uses text
representations to direct the perturbation of clean images, resulting in the
generation of adversarial images. FGA is orthogonal to many advanced attack
strategies in the unimodal domain, facilitating the direct application of rich
research findings from the unimodal to the multimodal scenario. By
appropriately introducing text attack into FGA, we construct Feature Guidance
with Text Attack (FGA-T). Through the interaction of attacking two modalities,
FGA-T achieves superior attack effects against VLP models. Moreover,
incorporating data augmentation and momentum mechanisms significantly improves
the black-box transferability of FGA-T. Our method demonstrates stable and
effective attack capabilities across various datasets, downstream tasks, and
both black-box and white-box settings, offering a unified baseline for
exploring the robustness of VLP models.
