  Text-to-image models encounter safety issues, including concerns related to
copyright and Not-Safe-For-Work (NSFW) content. Despite several methods have
been proposed for erasing inappropriate concepts from diffusion models, they
often exhibit incomplete erasure, consume a lot of computing resources, and
inadvertently damage generation ability. In this work, we introduce Reliable
and Efficient Concept Erasure (RECE), a novel approach that modifies the model
in 3 seconds without necessitating additional fine-tuning. Specifically, RECE
efficiently leverages a closed-form solution to derive new target embeddings,
which are capable of regenerating erased concepts within the unlearned model.
To mitigate inappropriate content potentially represented by derived
embeddings, RECE further aligns them with harmless concepts in cross-attention
layers. The derivation and erasure of new representation embeddings are
conducted iteratively to achieve a thorough erasure of inappropriate concepts.
Besides, to preserve the model's generation ability, RECE introduces an
additional regularization term during the derivation process, resulting in
minimizing the impact on unrelated concepts during the erasure process. All the
processes above are in closed-form, guaranteeing extremely efficient erasure in
only 3 seconds. Benchmarking against previous approaches, our method achieves
more efficient and thorough erasure with minor damage to original generation
ability and demonstrates enhanced robustness against red-teaming tools. Code is
available at \url{https://github.com/CharlesGong12/RECE}.
