  Self-supervised learning (SSL) is increasingly attractive for pre-training
encoders without requiring labeled data. Downstream tasks built on top of those
pre-trained encoders can achieve nearly state-of-the-art performance. The
pre-trained encoders by SSL, however, are vulnerable to backdoor attacks as
demonstrated by existing studies. Numerous backdoor mitigation techniques are
designed for downstream task models. However, their effectiveness is impaired
and limited when adapted to pre-trained encoders, due to the lack of label
information when pre-training. To address backdoor attacks against pre-trained
encoders, in this paper, we innovatively propose a mutual information guided
backdoor mitigation technique, named MIMIC. MIMIC treats the potentially
backdoored encoder as the teacher net and employs knowledge distillation to
distill a clean student encoder from the teacher net. Different from existing
knowledge distillation approaches, MIMIC initializes the student with random
weights, inheriting no backdoors from teacher nets. Then MIMIC leverages mutual
information between each layer and extracted features to locate where benign
knowledge lies in the teacher net, with which distillation is deployed to clone
clean features from teacher to student. We craft the distillation loss with two
aspects, including clone loss and attention loss, aiming to mitigate backdoors
and maintain encoder performance at the same time. Our evaluation conducted on
two backdoor attacks in SSL demonstrates that MIMIC can significantly reduce
the attack success rate by only utilizing <5% of clean data, surpassing seven
state-of-the-art backdoor mitigation techniques.
