  Many facts come with an expiration date, from the name of the President to
the basketball team Lebron James plays for. But language models (LMs) are
trained on snapshots of data collected at a specific moment in time, and this
can limit their utility, especially in the closed-book setting where the
pretraining corpus must contain the facts the model should memorize. We
introduce a diagnostic dataset aimed at probing LMs for factual knowledge that
changes over time and highlight problems with LMs at either end of the spectrum
-- those trained on specific slices of temporal data, as well as those trained
on a wide range of temporal data. To mitigate these problems, we propose a
simple technique for jointly modeling text with its timestamp. This improves
memorization of seen facts from the training time period, as well as
calibration on predictions about unseen facts from future time periods. We also
show that models trained with temporal context can be efficiently "refreshed"
as new data arrives, without the need for retraining from scratch.
