  Researchers have recently found that Self-Supervised Learning (SSL) is
vulnerable to backdoor attacks. The attacker can embed hidden SSL backdoors via
a few poisoned examples in the training dataset and maliciously manipulate the
behavior of downstream models. To defend against SSL backdoor attacks, a
feasible route is to detect and remove the poisonous samples in the training
set. However, the existing SSL backdoor defense method fails to detect the
poisonous samples precisely. In this paper, we propose to erase the SSL
backdoor by cluster activation masking and propose a novel PoisonCAM method.
After obtaining the threat model trained on the poisoned dataset, our method
can precisely detect poisonous samples based on the assumption that masking the
backdoor trigger can effectively change the activation of a downstream
clustering model. In experiments, our PoisonCAM achieves 96% accuracy for
backdoor trigger detection compared to 3% of the state-of-the-art method on
poisoned ImageNet-100. Moreover, our proposed PoisonCAM significantly improves
the performance of the trained SSL model under backdoor attacks compared to the
state-of-the-art method. Our code will be available at
https://github.com/LivXue/PoisonCAM.
