  While large-scale text-to-image diffusion models have demonstrated impressive
image-generation capabilities, there are significant concerns about their
potential misuse for generating unsafe content, violating copyright, and
perpetuating societal biases. Recently, the text-to-image generation community
has begun addressing these concerns by editing or unlearning undesired concepts
from pre-trained models. However, these methods often involve data-intensive
and inefficient fine-tuning or utilize various forms of token remapping,
rendering them susceptible to adversarial jailbreaks. In this paper, we present
a simple and effective training-free approach, ConceptPrune, wherein we first
identify critical regions within pre-trained models responsible for generating
undesirable concepts, thereby facilitating straightforward concept unlearning
via weight pruning. Experiments across a range of concepts including artistic
styles, nudity, object erasure, and gender debiasing demonstrate that target
concepts can be efficiently erased by pruning a tiny fraction, approximately
0.12% of total weights, enabling multi-concept erasure and robustness against
various white-box and black-box adversarial attacks.
