  Diffusion models (DMs) based adversarial purification (AP) has shown to be
the most powerful alternative to adversarial training (AT). However, these
methods neglect the fact that pre-trained diffusion models themselves are not
robust to adversarial attacks as well. Additionally, the diffusion process can
easily destroy semantic information and generate a high quality image but
totally different from the original input image after the reverse process,
leading to degraded standard accuracy. To overcome these issues, a natural idea
is to harness adversarial training strategy to retrain or fine-tune the
pre-trained diffusion model, which is computationally prohibitive. We propose a
novel robust reverse process with adversarial guidance, which is independent of
given pre-trained DMs and avoids retraining or fine-tuning the DMs. This robust
guidance can not only ensure to generate purified examples retaining more
semantic content but also mitigate the accuracy-robustness trade-off of DMs for
the first time, which also provides DM-based AP an efficient adaptive ability
to new attacks. Extensive experiments are conducted on CIFAR-10, CIFAR-100 and
ImageNet to demonstrate that our method achieves the state-of-the-art results
and exhibits generalization against different attacks.
