  Recent studies have revealed the vulnerability of deep neural networks (DNNs)
to various backdoor attacks, where the behavior of DNNs can be compromised by
utilizing certain types of triggers or poisoning mechanisms. State-of-the-art
(SOTA) defenses employ too-sophisticated mechanisms that require either a
computationally expensive adversarial search module for reverse-engineering the
trigger distribution or an over-sensitive hyper-parameter selection module.
Moreover, they offer sub-par performance in challenging scenarios, e.g.,
limited validation data and strong attacks. In this paper, we propose Neural
mask Fine-Tuning (NFT) with an aim to optimally re-organize the neuron
activities in a way that the effect of the backdoor is removed. Utilizing a
simple data augmentation like MixUp, NFT relaxes the trigger synthesis process
and eliminates the requirement of the adversarial search module. Our study
further reveals that direct weight fine-tuning under limited validation data
results in poor post-purification clean test accuracy, primarily due to
overfitting issue. To overcome this, we propose to fine-tune neural masks
instead of model weights. In addition, a mask regularizer has been devised to
further mitigate the model drift during the purification process. The distinct
characteristics of NFT render it highly efficient in both runtime and sample
usage, as it can remove the backdoor even when a single sample is available
from each class. We validate the effectiveness of NFT through extensive
experiments covering the tasks of image classification, object detection, video
action recognition, 3D point cloud, and natural language processing. We
evaluate our method against 14 different attacks (LIRA, WaNet, etc.) on 11
benchmark data sets such as ImageNet, UCF101, Pascal VOC, ModelNet,
OpenSubtitles2012, etc.
