  Retrieval-augmented generation (RAG) is considered to be a promising approach
to alleviate the hallucination issue of large language models (LLMs), and it
has received widespread attention from researchers recently. Due to the
limitation in the semantic understanding of retrieval models, the success of
RAG heavily lies on the ability of LLMs to identify passages with utility.
Recent efforts have explored the ability of LLMs to assess the relevance of
passages in retrieval, but there has been limited work on evaluating the
utility of passages in supporting question answering. In this work, we conduct
a comprehensive study about the capabilities of LLMs in utility evaluation for
open-domain QA. Specifically, we introduce a benchmarking procedure and
collection of candidate passages with different characteristics, facilitating a
series of experiments with five representative LLMs. Our experiments reveal
that: (i) well-instructed LLMs can distinguish between relevance and utility,
and that LLMs are highly receptive to newly generated counterfactual passages.
Moreover, (ii) we scrutinize key factors that affect utility judgments in the
instruction design. And finally, (iii) to verify the efficacy of utility
judgments in practical retrieval augmentation applications, we delve into LLMs'
QA capabilities using the evidence judged with utility and direct dense
retrieval results. (iv) We propose a k-sampling, listwise approach to reduce
the dependency of LLMs on the sequence of input passages, thereby facilitating
subsequent answer generation. We believe that the way we formalize and study
the problem along with our findings contributes to a critical assessment of
retrieval-augmented LLMs. Our code and benchmark can be found at
\url{https://github.com/ict-bigdatalab/utility_judgments}.
