  Text-to-image diffusion models can create realistic images based on input
texts. Users can describe an object to convey their opinions visually. In this
work, we unveil a previously unrecognized and latent risk of using diffusion
models to generate images; we utilize emotion in the input texts to introduce
negative contents, potentially eliciting unfavorable emotions in users.
Emotions play a crucial role in expressing personal opinions in our daily
interactions, and the inclusion of maliciously negative content can lead users
astray, exacerbating negative emotions. Specifically, we identify the
emotion-aware backdoor attack (EmoAttack) that can incorporate malicious
negative content triggered by emotional texts during image generation. We
formulate such an attack as a diffusion personalization problem to avoid
extensive model retraining and propose the EmoBooth. Unlike existing
personalization methods, our approach fine-tunes a pre-trained diffusion model
by establishing a mapping between a cluster of emotional words and a given
reference image containing malicious negative content. To validate the
effectiveness of our method, we built a dataset and conducted extensive
analysis and discussion about its effectiveness. Given consumers' widespread
use of diffusion models, uncovering this threat is critical for society.
