  The fabrication of visual misinformation on the web and social media has
increased exponentially with the advent of foundational text-to-image diffusion
models. Namely, Stable Diffusion inpainters allow the synthesis of maliciously
inpainted images of personal and private figures, and copyrighted contents,
also known as deepfakes. To combat such generations, a disruption framework,
namely Photoguard, has been proposed, where it adds adversarial noise to the
context image to disrupt their inpainting synthesis. While their framework
suggested a diffusion-friendly approach, the disruption is not sufficiently
strong and it requires a significant amount of GPU and time to immunize the
context image. In our work, we re-examine both the minimal and favorable
conditions for a successful inpainting disruption, proposing DDD, a "Digression
guided Diffusion Disruption" framework. First, we identify the most
adversarially vulnerable diffusion timestep range with respect to the hidden
space. Within this scope of noised manifold, we pose the problem as a semantic
digression optimization. We maximize the distance between the inpainting
instance's hidden states and a semantic-aware hidden state centroid, calibrated
both by Monte Carlo sampling of hidden states and a discretely projected
optimization in the token space. Effectively, our approach achieves stronger
disruption and a higher success rate than Photoguard while lowering the GPU
memory requirement, and speeding the optimization up to three times faster.
