  With the help of conditioning mechanisms, the state-of-the-art diffusion
models have achieved tremendous success in guided image generation,
particularly in text-to-image synthesis. To gain a better understanding of the
training process and potential risks of text-to-image synthesis, we perform a
systematic investigation of backdoor attack on text-to-image diffusion models
and propose BadT2I, a general multimodal backdoor attack framework that tampers
with image synthesis in diverse semantic levels. Specifically, we perform
backdoor attacks on three levels of the vision semantics: Pixel-Backdoor,
Object-Backdoor and Style-Backdoor. By utilizing a regularization loss, our
methods efficiently inject backdoors into a large-scale text-to-image diffusion
model while preserving its utility with benign inputs. We conduct empirical
experiments on Stable Diffusion, the widely-used text-to-image diffusion model,
demonstrating that the large-scale diffusion model can be easily backdoored
within a few fine-tuning steps. We conduct additional experiments to explore
the impact of different types of textual triggers, as well as the backdoor
persistence during further training, providing insights for the development of
backdoor defense methods. Besides, our investigation may contribute to the
copyright protection of text-to-image models in the future.
