  Backdoor attacks have emerged as an urgent threat to Deep Neural Networks
(DNNs), where victim DNNs are furtively implanted with malicious neurons that
could be triggered by the adversary. To defend against backdoor attacks, many
works establish a staged pipeline to remove backdoors from victim DNNs:
inspecting, locating, and erasing. However, in a scenario where a few clean
data can be accessible, such pipeline is fragile and cannot erase backdoors
completely without sacrificing model accuracy. To address this issue, in this
paper, we propose a novel data-free holistic backdoor erasing (DHBE) framework.
Instead of the staged pipeline, the DHBE treats the backdoor erasing task as a
unified adversarial procedure, which seeks equilibrium between two different
competing processes: distillation and backdoor regularization. In distillation,
the backdoored DNN is distilled into a proxy model, transferring its knowledge
about clean data, yet backdoors are simultaneously transferred. In backdoor
regularization, the proxy model is holistically regularized to prevent from
infecting any possible backdoor transferred from distillation. These two
processes jointly proceed with data-free adversarial optimization until a
clean, high-accuracy proxy model is obtained. With the novel adversarial
design, our framework demonstrates its superiority in three aspects: 1) minimal
detriment to model accuracy, 2) high tolerance for hyperparameters, and 3) no
demand for clean data. Extensive experiments on various backdoor attacks and
datasets are performed to verify the effectiveness of the proposed framework.
Code is available at \url{https://github.com/yanzhicong/DHBE}
