  This paper addresses the societal concerns arising from large-scale
text-to-image diffusion models for generating potentially harmful or
copyrighted content. Existing models rely heavily on internet-crawled data,
wherein problematic concepts persist due to incomplete filtration processes.
While previous approaches somewhat alleviate the issue, they often rely on
text-specified concepts, introducing challenges in accurately capturing nuanced
concepts and aligning model knowledge with human understandings. In response,
we propose a framework named Human Feedback Inversion (HFI), where human
feedback on model-generated images is condensed into textual tokens guiding the
mitigation or removal of problematic images. The proposed framework can be
built upon existing techniques for the same purpose, enhancing their alignment
with human judgment. By doing so, we simplify the training objective with a
self-distillation-based technique, providing a strong baseline for concept
removal. Our experimental results demonstrate our framework significantly
reduces objectionable content generation while preserving image quality,
contributing to the ethical deployment of AI in the public sphere.
