  Language models (LMs) are typically trained once on a large-scale corpus and
used for years without being updated. However, in a dynamic world, new entities
constantly arise. We propose a framework to analyze what LMs can infer about
new entities that did not exist when the LMs were pretrained. We derive a
dataset of entities indexed by their origination date and paired with their
English Wikipedia articles, from which we can find sentences about each entity.
We evaluate LMs' perplexity on masked spans within these sentences. We show
that models more informed about the entities, such as those with access to a
textual definition of them, achieve lower perplexity on this benchmark. Our
experimental results demonstrate that making inferences about new entities
remains difficult for LMs. Given its wide coverage on entity knowledge and
temporal indexing, our dataset can be used to evaluate LMs and techniques
designed to modify or extend their knowledge. Our automatic data collection
pipeline can be easily used to continually update our benchmark.
