  The information stored in large language models (LLMs) falls out of date
quickly, and retraining from scratch is often not an option. This has recently
given rise to a range of techniques for injecting new facts through updating
model weights. Current evaluation paradigms are extremely limited, mainly
validating the recall of edited facts, but changing one fact should cause
rippling changes to the model's related beliefs. If we edit the UK Prime
Minister to now be Rishi Sunak, then we should get a different answer to Who is
married to the British Prime Minister? In this work, we present a benchmark,
MQuAKE (Multi-hop Question Answering for Knowledge Editing), comprising
multi-hop questions that assess whether edited models correctly answer
questions where the answer should change as an entailed consequence of edited
facts. While we find that current knowledge-editing approaches can recall
edited facts accurately, they fail catastrophically on the constructed
multi-hop questions. We thus propose a simple memory-based approach, MeLLo,
which stores all edited facts externally while prompting the language model
iteratively to generate answers that are consistent with the edited facts.
While MQuAKE remains challenging, we show that MeLLo scales well with LLMs (up
to 175B) and outperforms previous model editors by a large margin.
