  Large language models (LLMs) inevitably memorize sensitive, copyrighted, and
harmful knowledge from the training corpus; therefore, it is crucial to erase
this knowledge from the models. Machine unlearning is a promising solution for
efficiently removing specific knowledge by post hoc modifying models. In this
paper, we propose a Real-World Knowledge Unlearning benchmark (RWKU) for LLM
unlearning. RWKU is designed based on the following three key factors: (1) For
the task setting, we consider a more practical and challenging unlearning
setting, where neither the forget corpus nor the retain corpus is accessible.
(2) For the knowledge source, we choose 200 real-world famous people as the
unlearning targets and show that such popular knowledge is widely present in
various LLMs. (3) For the evaluation framework, we design the forget set and
the retain set to evaluate the model's capabilities across various real-world
applications. Regarding the forget set, we provide four four membership
inference attack (MIA) methods and nine kinds of adversarial attack probes to
rigorously test unlearning efficacy. Regarding the retain set, we assess
locality and utility in terms of neighbor perturbation, general ability,
reasoning ability, truthfulness, factuality, and fluency. We conduct extensive
experiments across two unlearning scenarios, two models and six baseline
methods and obtain some meaningful findings. We release our benchmark and code
publicly at http://rwku-bench.github.io for future work.
