  LLMs are known to be vulnerable to jailbreak attacks, even after safety
alignment. An important observation is that, while different types of jailbreak
attacks can generate significantly different queries, they mostly result in
similar responses that are rooted in the same harmful knowledge (e.g., detailed
steps to make a bomb). Therefore, we conjecture that directly unlearn the
harmful knowledge in the LLM can be a more effective way to defend against
jailbreak attacks than the mainstream supervised fine-tuning (SFT) based
approaches. Our extensive experiments confirmed our insight and suggested
surprising generalizability of our unlearning-based approach: using only 20 raw
harmful questions \emph{without} any jailbreak prompt during training, our
solution reduced the Attack Success Rate (ASR) in Vicuna-7B on
\emph{out-of-distribution} (OOD) harmful questions wrapped with various complex
jailbreak prompts from 82.6\% to 7.7\%. This significantly outperforms
Llama2-7B-Chat, which is fine-tuned on about 0.1M safety alignment samples but
still has an ASR of 21.9\% even under the help of an additional safety system
prompt. Further analysis reveals that the generalization ability of our
solution stems from the intrinsic relatedness among harmful responses across
harmful questions (e.g., response patterns, shared steps and actions, and
similarity among their learned representations in the LLM). Our code is
available at \url{https://github.com/thu-coai/SafeUnlearning}.
