  The factuality of large language model (LLMs) tends to decay over time since
events posterior to their training are "unknown" to them. One way to keep
models up-to-date could be factual update: the task of inserting, replacing, or
removing certain simple (atomic) facts within the model. To study this task, we
present WikiFactDiff, a dataset that describes the evolution of factual
knowledge between two dates as a collection of simple facts divided into three
categories: new, obsolete, and static. We describe several update scenarios
arising from various combinations of these three types of basic update. The
facts are represented by subject-relation-object triples; indeed, WikiFactDiff
was constructed by comparing the state of the Wikidata knowledge base at 4
January 2021 and 27 February 2023. Those fact are accompanied by verbalization
templates and cloze tests that enable running update algorithms and their
evaluation metrics. Contrary to other datasets, such as zsRE and CounterFact,
WikiFactDiff constitutes a realistic update setting that involves various
update scenarios, including replacements, archival, and new entity insertions.
We also present an evaluation of existing update algorithms on WikiFactDiff.
