  Model editing aims to correct outdated or erroneous knowledge in large
language models (LLMs) without the need for costly retraining. Lifelong model
editing is the most challenging task that caters to the continuous editing
requirements of LLMs. Prior works primarily focus on single or batch editing;
nevertheless, these methods fall short in lifelong editing scenarios due to
catastrophic knowledge forgetting and the degradation of model performance.
Although retrieval-based methods alleviate these issues, they are impeded by
slow and cumbersome processes of integrating the retrieved knowledge into the
model. In this work, we introduce RECIPE, a RetriEval-augmented ContInuous
Prompt lEarning method, to boost editing efficacy and inference efficiency in
lifelong learning. RECIPE first converts knowledge statements into short and
informative continuous prompts, prefixed to the LLM's input query embedding, to
efficiently refine the response grounded on the knowledge. It further
integrates the Knowledge Sentinel (KS) that acts as an intermediary to
calculate a dynamic threshold, determining whether the retrieval repository
contains relevant knowledge. Our retriever and prompt encoder are jointly
trained to achieve editing properties, i.e., reliability, generality, and
locality. In our experiments, RECIPE is assessed extensively across multiple
LLMs and editing datasets, where it achieves superior editing performance.
RECIPE also demonstrates its capability to maintain the overall performance of
LLMs alongside showcasing fast editing and inference speed.
