  The Internet contains a wealth of knowledge -- from the birthdays of
historical figures to tutorials on how to code -- all of which may be learned
by language models. However, while certain pieces of information are ubiquitous
on the web, others appear extremely rarely. In this paper, we study the
relationship between the knowledge memorized by large language models and the
information in pre-training datasets scraped from the web. In particular, we
show that a language model's ability to answer a fact-based question relates to
how many documents associated with that question were seen during pre-training.
We identify these relevant documents by entity linking pre-training datasets
and counting documents that contain the same entities as a given
question-answer pair. Our results demonstrate strong correlational and causal
relationships between accuracy and relevant document count for numerous
question answering datasets (e.g., TriviaQA), pre-training corpora (e.g.,
ROOTS), and model sizes (e.g., 176B parameters). Moreover, while larger models
are better at learning long-tail knowledge, we estimate that today's models
must be scaled by many orders of magnitude to reach competitive QA performance
on questions with little support in the pre-training data. Finally, we show
that retrieval-augmentation can reduce the dependence on relevant pre-training
information, presenting a promising approach for capturing the long-tail.
