  Adaptive retrieval-augmented generation (ARAG) aims to dynamically determine
the necessity of retrieval for queries instead of retrieving indiscriminately
to enhance the efficiency and relevance of the sourced information. However,
previous works largely overlook the evaluation of ARAG approaches, leading to
their effectiveness being understudied. This work presents a benchmark,
RetrievalQA, comprising 1,271 short-form questions covering new world and
long-tail knowledge. The knowledge necessary to answer the questions is absent
from LLMs; therefore, external information must be retrieved to answer
correctly. This makes RetrievalQA a suitable testbed to evaluate existing ARAG
methods. We observe that calibration-based methods heavily rely on threshold
tuning, while vanilla prompting is inadequate for guiding LLMs to make reliable
retrieval decisions. Based on our findings, we propose Time-Aware Adaptive
Retrieval (TA-ARE), a simple yet effective method that helps LLMs assess the
necessity of retrieval without calibration or additional training. The dataset
and code will be available at https://github.com/hyintell/RetrievalQA
