  Retrieval-Augmented Generation (RAG) has become a standard architectural
pattern for incorporating domain-specific knowledge into user-facing chat
applications powered by Large Language Models (LLMs). RAG systems are
characterized by (1) a document retriever that queries a domain-specific corpus
for context information relevant to an input query, and (2) an LLM that
generates a response based on the provided query and context. However,
comprehensive evaluation of RAG systems remains a challenge due to the lack of
unified evaluation criteria and annotated datasets. In response, we introduce
RAGBench: the first comprehensive, large-scale RAG benchmark dataset of 100k
examples. It covers five unique industry-specific domains and various RAG task
types. RAGBench examples are sourced from industry corpora such as user
manuals, making it particularly relevant for industry applications. Further, we
formalize the TRACe evaluation framework: a set of explainable and actionable
RAG evaluation metrics applicable across all RAG domains. We release the
labeled dataset at https://huggingface.co/datasets/rungalileo/ragbench.
RAGBench explainable labels facilitate holistic evaluation of RAG systems,
enabling actionable feedback for continuous improvement of production
applications. Thorough extensive benchmarking, we find that LLM-based RAG
evaluation methods struggle to compete with a finetuned RoBERTa model on the
RAG evaluation task. We identify areas where existing approaches fall short and
propose the adoption of RAGBench with TRACe towards advancing the state of RAG
evaluation systems.
