  The proliferation of large AI models trained on uncurated, often sensitive
web-scraped data has raised significant privacy concerns. One of the concerns
is that adversaries can extract information about the training data using
privacy attacks. Unfortunately, the task of removing specific information from
the models without sacrificing performance is not straightforward and has
proven to be challenging. We propose a rather easy yet effective defense based
on backdoor attacks to remove private information, such as names and faces of
individuals, from vision-language models by fine-tuning them for only a few
minutes instead of re-training them from scratch. Specifically, by
strategically inserting backdoors into text encoders, we align the embeddings
of sensitive phrases with those of neutral terms-"a person" instead of the
person's actual name. For image encoders, we map individuals' embeddings to be
removed from the model to a universal, anonymous embedding. The results of our
extensive experimental evaluation demonstrate the effectiveness of our
backdoor-based defense on CLIP by assessing its performance using a specialized
privacy attack for zero-shot classifiers. Our approach provides a new
"dual-use" perspective on backdoor attacks and presents a promising avenue to
enhance the privacy of individuals within models trained on uncurated
web-scraped data.
