  Recent developments in Language Models (LMs) have shown their effectiveness
in NLP tasks, particularly in knowledge-intensive tasks. However, the
mechanisms underlying knowledge storage and memory access within their
parameters remain elusive. In this paper, we investigate whether a generative
LM (e.g., GPT-2) is able to access its memory sequentially or randomly. Through
carefully-designed synthetic tasks, covering the scenarios of full recitation,
selective recitation and grounded question answering, we reveal that LMs manage
to sequentially access their memory while encountering challenges in randomly
accessing memorized content. We find that techniques including recitation and
permutation improve the random memory access capability of LMs. Furthermore, by
applying this intervention to realistic scenarios of open-domain question
answering, we validate that enhancing random access by recitation leads to
notable improvements in question answering. The code to reproduce our
experiments can be found at https://github.com/sail-sg/lm-random-memory-access.
