  Despite the advanced capabilities of contemporary machine learning (ML)
models, they remain vulnerable to adversarial and backdoor attacks. This
vulnerability is particularly concerning in real-world deployments, where
compromised models may exhibit unpredictable behavior in critical scenarios.
Such risks are heightened by the prevalent practice of collecting massive,
internet-sourced datasets for pre-training multimodal models, as these datasets
may harbor backdoors. Various techniques have been proposed to mitigate the
effects of backdooring in these models such as CleanCLIP which is the current
state-of-the-art approach. In this work, we demonstrate that the efficacy of
CleanCLIP in mitigating backdoors is highly dependent on the particular
objective used during model pre-training. We observe that stronger pre-training
objectives correlate with harder to remove backdoors behaviors. We show this by
training multimodal models on two large datasets consisting of 3 million (CC3M)
and 6 million (CC6M) datapoints, under various pre-training objectives,
followed by poison removal using CleanCLIP. We find that CleanCLIP is
ineffective when stronger pre-training objectives are used, even with extensive
hyperparameter tuning. Our findings underscore critical considerations for ML
practitioners who pre-train models using large-scale web-curated data and are
concerned about potential backdoor threats. Notably, our results suggest that
simpler pre-training objectives are more amenable to effective backdoor
removal. This insight is pivotal for practitioners seeking to balance the
trade-offs between using stronger pre-training objectives and security against
backdoor attacks.
