  Large language models show impressive abilities in memorizing world
knowledge, which leads to concerns regarding memorization of private
information, toxic or sensitive knowledge, and copyrighted content. We
introduce the problem of Large Scale Knowledge Washing, focusing on unlearning
an extensive amount of factual knowledge. Previous unlearning methods usually
define the reverse loss and update the model via backpropagation, which may
affect the model's fluency and reasoning ability or even destroy the model due
to extensive training with the reverse loss. Existing works introduce
additional data from downstream tasks to prevent the model from losing
capabilities, which requires downstream task awareness. Controlling the
tradeoff of unlearning and maintaining existing capabilities is also
challenging. To this end, we propose LAW (Large Scale Washing) to update the
MLP layers in decoder-only large language models to perform knowledge washing,
as inspired by model editing methods and based on the hypothesis that knowledge
and reasoning are disentanglable. We derive a new objective with the knowledge
to be unlearned to update the weights of certain MLP layers. Experimental
results demonstrate the effectiveness of LAW in forgetting target knowledge
while maintaining reasoning ability. The code will be open-sourced at
https://github.com/wangyu-ustc/LargeScaleWashing.
