  Adversarial attacks, particularly patch attacks, pose significant threats to
the robustness and reliability of deep learning models. Developing reliable
defenses against patch attacks is crucial for real-world applications. This
paper introduces DIFFender, a novel defense framework that harnesses the
capabilities of a text-guided diffusion model to combat patch attacks. Central
to our approach is the discovery of the Adversarial Anomaly Perception (AAP)
phenomenon, which empowers the diffusion model to detect and localize
adversarial patches through the analysis of distributional discrepancies.
DIFFender integrates dual tasks of patch localization and restoration within a
single diffusion model framework, utilizing their close interaction to enhance
defense efficacy. Moreover, DIFFender utilizes vision-language pre-training
coupled with an efficient few-shot prompt-tuning algorithm, which streamlines
the adaptation of the pre-trained diffusion model to defense tasks, thus
eliminating the need for extensive retraining. Our comprehensive evaluation
spans image classification and face recognition tasks, extending to real-world
scenarios, where DIFFender shows good robustness against adversarial attacks.
The versatility and generalizability of DIFFender are evident across a variety
of settings, classifiers, and attack methodologies, marking an advancement in
adversarial patch defense strategies.
