  Challenges in the automated evaluation of Retrieval-Augmented Generation
(RAG) Question-Answering (QA) systems include hallucination problems in
domain-specific knowledge and the lack of gold standard benchmarks for company
internal tasks. This results in difficulties in evaluating RAG variations, like
RAG-Fusion (RAGF), in the context of a product QA task at Infineon
Technologies. To solve these problems, we propose a comprehensive evaluation
framework, which leverages Large Language Models (LLMs) to generate large
datasets of synthetic queries based on real user queries and in-domain
documents, uses LLM-as-a-judge to rate retrieved documents and answers,
evaluates the quality of answers, and ranks different variants of
Retrieval-Augmented Generation (RAG) agents with RAGElo's automated Elo-based
competition. LLM-as-a-judge rating of a random sample of synthetic queries
shows a moderate, positive correlation with domain expert scoring in relevance,
accuracy, completeness, and precision. While RAGF outperformed RAG in Elo
score, a significance analysis against expert annotations also shows that RAGF
significantly outperforms RAG in completeness, but underperforms in precision.
In addition, Infineon's RAGF assistant demonstrated slightly higher performance
in document relevance based on MRR@5 scores. We find that RAGElo positively
aligns with the preferences of human annotators, though due caution is still
required. Finally, RAGF's approach leads to more complete answers based on
expert annotations and better answers overall based on RAGElo's evaluation
criteria.
