  Recently, self-supervised learning (SSL) was shown to be vulnerable to
patch-based data poisoning backdoor attacks. It was shown that an adversary can
poison a small part of the unlabeled data so that when a victim trains an SSL
model on it, the final model will have a backdoor that the adversary can
exploit. This work aims to defend self-supervised learning against such
attacks. We use a three-step defense pipeline, where we first train a model on
the poisoned data. In the second step, our proposed defense algorithm
(PatchSearch) uses the trained model to search the training data for poisoned
samples and removes them from the training set. In the third step, a final
model is trained on the cleaned-up training set. Our results show that
PatchSearch is an effective defense. As an example, it improves a model's
accuracy on images containing the trigger from 38.2% to 63.7% which is very
close to the clean model's accuracy, 64.6%. Moreover, we show that PatchSearch
outperforms baselines and state-of-the-art defense approaches including those
using additional clean, trusted data. Our code is available at
https://github.com/UCDvision/PatchSearch
