  Retrieval-Augmented Generation (RAG) has recently gained traction in natural
language processing. Numerous studies and real-world applications are
leveraging its ability to enhance generative models through external
information retrieval. Evaluating these RAG systems, however, poses unique
challenges due to their hybrid structure and reliance on dynamic knowledge
sources. To better understand these challenges, we conduct A Unified Evaluation
Process of RAG (Auepora) and aim to provide a comprehensive overview of the
evaluation and benchmarks of RAG systems. Specifically, we examine and compare
several quantifiable metrics of the Retrieval and Generation components, such
as relevance, accuracy, and faithfulness, within the current RAG benchmarks,
encompassing the possible output and ground truth pairs. We then analyze the
various datasets and metrics, discuss the limitations of current benchmarks,
and suggest potential directions to advance the field of RAG benchmarks.
