  Personalized diffusion models have gained popularity for adapting pre-trained
text-to-image models to generate images of specific topics with minimal
training data. However, these models are vulnerable to minor adversarial
perturbations, leading to degraded performance on corrupted datasets. Such
vulnerabilities are further exploited to craft protective perturbations on
sensitive images like portraits that prevent unauthorized generation. In
response, diffusion-based purification methods have been proposed to remove
these perturbations and retain generation performance. However, existing works
turn to over-purifying the images, which causes information loss. In this
paper, we take a closer look at the fine-tuning process of personalized
diffusion models through the lens of shortcut learning. And we propose a
hypothesis explaining the manipulation mechanisms of existing perturbation
methods, demonstrating that perturbed images significantly deviate from their
original prompts in the CLIP-based latent space. This misalignment during
fine-tuning causes models to associate noisy patterns with identifiers,
resulting in performance degradation. Based on these insights, we introduce a
systematic approach to maintain training performance through purification. Our
method first purifies the images to realign them with their original semantic
meanings in latent space. Then, we introduce contrastive learning with negative
tokens to decouple the learning of clean identities from noisy patterns, which
shows a strong potential capacity against adaptive perturbation. Our study
uncovers shortcut learning vulnerabilities in personalized diffusion models and
provides a firm evaluation framework for future protective perturbation
research. Code is available at https://github.com/liuyixin-louis/DiffShortcut.
