  While large language models (LMs) demonstrate remarkable performance, they
encounter challenges in providing accurate responses when queried for
information beyond their pre-trained memorization. Although augmenting them
with relevant external information can mitigate these issues, failure to
consider the necessity of retrieval may adversely affect overall performance.
Previous research has primarily focused on examining how entities influence
retrieval models and knowledge recall in LMs, leaving other aspects relatively
unexplored. In this work, our goal is to offer a more detailed, fact-centric
analysis by exploring the effects of combinations of entities and relations. To
facilitate this, we construct a new question answering (QA) dataset called
WiTQA (Wikipedia Triple Question Answers). This dataset includes questions
about entities and relations of various popularity levels, each accompanied by
a supporting passage. Our extensive experiments with diverse LMs and retrievers
reveal when retrieval does not consistently enhance LMs from the viewpoints of
fact-centric popularity. Confirming earlier findings, we observe that larger
LMs excel in recalling popular facts. However, they notably encounter
difficulty with infrequent entity-relation pairs compared to retrievers.
Interestingly, they can effectively retain popular relations of less common
entities. We demonstrate the efficacy of our finer-grained metric and insights
through an adaptive retrieval system that selectively employs retrieval and
recall based on the frequencies of entities and relations in the question.
