  In the evolving landscape of text-to-image (T2I) diffusion models, the
remarkable capability to generate high-quality images from textual descriptions
faces challenges with the potential misuse of reproducing sensitive content. To
address this critical issue, we introduce \textbf{R}obust \textbf{A}dversarial
\textbf{C}oncept \textbf{E}rase (RACE), a novel approach designed to mitigate
these risks by enhancing the robustness of concept erasure method for T2I
models. RACE utilizes a sophisticated adversarial training framework to
identify and mitigate adversarial text embeddings, significantly reducing the
Attack Success Rate (ASR). Impressively, RACE achieves a 30 percentage point
reduction in ASR for the ``nudity'' concept against the leading white-box
attack method. Our extensive evaluations demonstrate RACE's effectiveness in
defending against both white-box and black-box attacks, marking a significant
advancement in protecting T2I diffusion models from generating inappropriate or
misleading imagery. This work underlines the essential need for proactive
defense measures in adapting to the rapidly advancing field of adversarial
challenges. Our code is publicly available:
\url{https://github.com/chkimmmmm/R.A.C.E.}
