  Adversarial examples for diffusion models are widely used as solutions for
safety concerns. By adding adversarial perturbations to personal images,
attackers can not edit or imitate them easily. However, it is essential to note
that all these protections target the latent diffusion model (LDMs), the
adversarial examples for diffusion models in the pixel space (PDMs) are largely
overlooked. This may mislead us to think that the diffusion models are
vulnerable to adversarial attacks like most deep models. In this paper, we show
novel findings that: even though gradient-based white-box attacks can be used
to attack the LDMs, they fail to attack PDMs. This finding is supported by
extensive experiments of almost a wide range of attacking methods on various
PDMs and LDMs with different model structures, which means diffusion models are
indeed much more robust against adversarial attacks. We also find that PDMs can
be used as an off-the-shelf purifier to effectively remove the adversarial
patterns that were generated on LDMs to protect the images, which means that
most protection methods nowadays, to some extent, cannot protect our images
from malicious attacks. We hope that our insights will inspire the community to
rethink the adversarial samples for diffusion models as protection methods and
move forward to more effective protection. Codes are available in
https://github.com/xavihart/PDM-Pure.
