  Given the prevalence of large language models (LLMs) and the prohibitive cost
of training these models from scratch, dynamically forgetting specific
knowledge e.g., private or proprietary, without retraining the model has become
an important capability. This paper proposes a novel method to achieve this
objective called UNLEARN. The approach builds upon subspace methods to identify
and specifically target the removal of knowledge without adversely affecting
other knowledge in the LLM. Results demonstrate 96% of targeted knowledge can
be forgotten while maintaining performance on other knowledge within 2.5% of
the original model, significantly outperforming the discriminatory abilities of
the previous state-of-the-art. A dual method called LEARN is also proposed for
targeted knowledge addition. Results show LEARN can match the fine-tuning
accuracy of Low-Rank Adaptation (LoRA) without adversely affecting similar
tasks.
