  Pre-trained Large Language Models (LLMs) have demonstrated remarkable
capabilities but also pose risks by learning and generating copyrighted
material, leading to significant legal and ethical concerns. To address these
issues, it is critical for model owners to be able to unlearn copyrighted
content at various time steps. We explore the setting of sequential unlearning,
where copyrighted content is removed over multiple time steps - a scenario that
has not been rigorously addressed. To tackle this challenge, we propose Stable
Sequential Unlearning (SSU), a novel unlearning framework for LLMs, designed to
have a more stable process to remove copyrighted content from LLMs throughout
different time steps using task vectors, by incorporating additional random
labeling loss and applying gradient-based weight saliency mapping. Experiments
demonstrate that SSU finds a good balance between unlearning efficacy and
maintaining the model's general knowledge compared to existing baselines.
