  Contrastive vision-language representation learning has achieved
state-of-the-art performance for zero-shot classification, by learning from
millions of image-caption pairs crawled from the internet. However, the massive
data that powers large multimodal models such as CLIP, makes them extremely
vulnerable to various types of targeted data poisoning and backdoor attacks.
Despite this vulnerability, robust contrastive vision-language pre-training
against such attacks has remained unaddressed. In this work, we propose ROCLIP,
the first effective method for robust pre-training multimodal vision-language
models against targeted data poisoning and backdoor attacks. ROCLIP effectively
breaks the association between poisoned image-caption pairs by considering a
relatively large and varying pool of random captions, and matching every image
with the text that is most similar to it in the pool instead of its own
caption, every few epochs.It also leverages image and text augmentations to
further strengthen the defense and improve the performance of the model. Our
extensive experiments show that ROCLIP renders state-of-the-art targeted data
poisoning and backdoor attacks ineffective during pre-training CLIP models. In
particular, ROCLIP decreases the success rate for targeted data poisoning
attacks from 93.75% to 12.5% and that of backdoor attacks down to 0%, while
improving the model's linear probe performance by 10% and maintains a similar
zero shot performance compared to CLIP. By increasing the frequency of
matching, ROCLIP is able to defend strong attacks, which add up to 1% poisoned
examples to the data, and successfully maintain a low attack success rate of
12.5%, while trading off the performance on some tasks.
