  The advent of Large Language Models (LLMs) has significantly transformed the
AI landscape, enhancing machine learning and AI capabilities. Factuality issue
is a critical concern for LLMs, as they may generate factually incorrect
responses. In this paper, we propose GraphEval to evaluate an LLM's performance
using a substantially large test dataset. Specifically, the test dataset is
retrieved from a large knowledge graph with more than 10 million facts without
expensive human efforts. Unlike conventional methods that evaluate LLMs based
on generated responses, GraphEval streamlines the evaluation process by
creating a judge model to estimate the correctness of the answers given by the
LLM. Our experiments demonstrate that the judge model's factuality assessment
aligns closely with the correctness of the LLM's generated outputs, while also
substantially reducing evaluation costs. Besides, our findings offer valuable
insights into LLM performance across different metrics and highlight the
potential for future improvements in ensuring the factual integrity of LLM
outputs. The code is publicly available at https://github.com/xz-liu/GraphEval.
