  Diffusion-based personalized visual content generation technologies have
achieved significant breakthroughs, allowing for the creation of specific
objects by just learning from a few reference photos. However, when misused to
fabricate fake news or unsettling content targeting individuals, these
technologies could cause considerable societal harm. To address this problem,
current methods generate adversarial samples by adversarially maximizing the
training loss, thereby disrupting the output of any personalized generation
model trained with these samples. However, the existing methods fail to achieve
effective defense and maintain stealthiness, as they overlook the intrinsic
properties of diffusion models. In this paper, we introduce a novel Dual-Domain
Anti-Personalization framework (DDAP). Specifically, we have developed Spatial
Perturbation Learning (SPL) by exploiting the fixed and perturbation-sensitive
nature of the image encoder in personalized generation. Subsequently, we have
designed a Frequency Perturbation Learning (FPL) method that utilizes the
characteristics of diffusion models in the frequency domain. The SPL disrupts
the overall texture of the generated images, while the FPL focuses on image
details. By alternating between these two methods, we construct the DDAP
framework, effectively harnessing the strengths of both domains. To further
enhance the visual quality of the adversarial samples, we design a localization
module to accurately capture attentive areas while ensuring the effectiveness
of the attack and avoiding unnecessary disturbances in the background.
Extensive experiments on facial benchmarks have shown that the proposed DDAP
enhances the disruption of personalized generation models while also
maintaining high quality in adversarial samples, making it more effective in
protecting privacy in practical applications.
