  Recent advances in Large Language Models (LLMs) have significantly shaped the
applications of AI in multiple fields, including the studies of legal
intelligence. Trained on extensive legal texts, including statutes and legal
documents, the legal LLMs can capture important legal knowledge/concepts
effectively and provide important support for downstream legal applications
such as legal consultancy. Yet, the dynamic nature of legal statutes and
interpretations also poses new challenges to the use of LLMs in legal
applications. Particularly, how to update the legal knowledge of LLMs
effectively and efficiently has become an important research problem in
practice. Existing benchmarks for evaluating knowledge update methods are
mostly designed for the open domain and cannot address the specific challenges
of the legal domain, such as the nuanced application of new legal knowledge,
the complexity and lengthiness of legal regulations, and the intricate nature
of legal reasoning. To address this gap, we introduce the Legal Knowledge
Update BEnchmark, i.e. LeKUBE, which evaluates knowledge update methods for
legal LLMs across five dimensions. Specifically, we categorize the needs of
knowledge updates in the legal domain with the help of legal professionals, and
then hire annotators from law schools to create synthetic updates to the
Chinese Criminal and Civil Code as well as sets of questions of which the
answers would change after the updates. Through a comprehensive evaluation of
state-of-the-art knowledge update methods, we reveal a notable gap between
existing knowledge update methods and the unique needs of the legal domain,
emphasizing the need for further research and development of knowledge update
mechanisms tailored for legal LLMs.
