  What do language models (LMs) do with language? Everyone agrees that they can
produce sequences of (mostly) coherent strings of English. But do those
sentences mean something, or are LMs simply babbling in a convincing simulacrum
of language use? Here we will address one aspect of this broad question:
whether LMs' words can refer, that is, achieve "word-to-world" connections.
There is prima facie reason to think they do not since LMs do not interact with
the world in the way that ordinary language users do. Drawing on insights from
the externalist tradition in philosophy of language, we argue that those
appearances are misleading: even if the inputs to an LM are simply strings of
text, they are strings of text with natural histories, and that may suffice to
put LMs' words into referential contact with the external world.
