  Large language models (LLMs) often make factually incorrect responses despite
their success in various applications. In this paper, we hypothesize that
relying heavily on simple co-occurrence statistics of the pre-training corpora
is one of the main factors that cause factual errors. Our results reveal that
LLMs are vulnerable to the co-occurrence bias, defined as preferring frequently
co-occurred words over the correct answer. Consequently, LLMs struggle to
recall facts whose subject and object rarely co-occur in the pre-training
dataset although they are seen during finetuning. We show that co-occurrence
bias remains despite scaling up model sizes or finetuning. Therefore, we
suggest finetuning on a debiased dataset to mitigate the bias by filtering out
biased samples whose subject-object co-occurrence count is high. Although
debiased finetuning allows LLMs to memorize rare facts in the training set, it
is not effective in recalling rare facts unseen during finetuning. Further
research in mitigation will help build reliable language models by preventing
potential errors. The code is available at
\url{https://github.com/CheongWoong/impact_of_cooccurrence}.
