  Multimodal contrastive learning methods (e.g., CLIP) have shown impressive
zero-shot classification performance due to their strong ability to joint
representation learning for visual and textual modalities. However, recent
research revealed that multimodal contrastive learning on poisoned pre-training
data with a small proportion of maliciously backdoored data can induce
backdoored CLIP that could be attacked by inserted triggers in downstream tasks
with a high success rate. To defend against backdoor attacks on CLIP, existing
defense methods focus on either the pre-training stage or the fine-tuning
stage, which would unfortunately cause high computational costs due to numerous
parameter updates. In this paper, we provide the first attempt at a
computationally efficient backdoor detection method to defend against
backdoored CLIP in the inference stage. We empirically find that the visual
representations of backdoored images are insensitive to both benign and
malignant changes in class description texts. Motivated by this observation, we
propose BDetCLIP, a novel test-time backdoor detection method based on
contrastive prompting. Specifically, we first prompt the language model (e.g.,
GPT-4) to produce class-related description texts (benign) and class-perturbed
random texts (malignant) by specially designed instructions. Then, the
distribution difference in cosine similarity between images and the two types
of class description texts can be used as the criterion to detect backdoor
samples. Extensive experiments validate that our proposed BDetCLIP is superior
to state-of-the-art backdoor detection methods, in terms of both effectiveness
and efficiency.
