  Recently, the newly emerged multimodal models, which leverage both visual and
linguistic modalities to train powerful encoders, have gained increasing
attention. However, learning from a large-scale unlabeled dataset also exposes
the model to the risk of potential poisoning attacks, whereby the adversary
aims to perturb the model's training data to trigger malicious behaviors in it.
In contrast to previous work, only poisoning visual modality, in this work, we
take the first step to studying poisoning attacks against multimodal models in
both visual and linguistic modalities. Specially, we focus on answering two
questions: (1) Is the linguistic modality also vulnerable to poisoning attacks?
and (2) Which modality is most vulnerable? To answer the two questions, we
propose three types of poisoning attacks against multimodal models. Extensive
evaluations on different datasets and model architectures show that all three
attacks can achieve significant attack performance while maintaining model
utility in both visual and linguistic modalities. Furthermore, we observe that
the poisoning effect differs between different modalities. To mitigate the
attacks, we propose both pre-training and post-training defenses. We
empirically show that both defenses can significantly reduce the attack
performance while preserving the model's utility.
