  The capabilities of large language models (LLMs) have sparked debate over
whether such systems just learn an enormous collection of superficial
statistics or a set of more coherent and grounded representations that reflect
the real world. We find evidence for the latter by analyzing the learned
representations of three spatial datasets (world, US, NYC places) and three
temporal datasets (historical figures, artworks, news headlines) in the Llama-2
family of models. We discover that LLMs learn linear representations of space
and time across multiple scales. These representations are robust to prompting
variations and unified across different entity types (e.g. cities and
landmarks). In addition, we identify individual "space neurons" and "time
neurons" that reliably encode spatial and temporal coordinates. While further
investigation is needed, our results suggest modern LLMs learn rich
spatiotemporal representations of the real world and possess basic ingredients
of a world model.
