  In this paper, we initiate our discussion by demonstrating how Large Language
Models (LLMs), when tasked with responding to queries, display a more even
probability distribution in their answers if they are more adept, as opposed to
their less skilled counterparts. Expanding on this foundational insight, we
propose a new self-evaluation method ProbDiff for assessing the efficacy of
various LLMs. This approach obviates the necessity for an additional evaluation
model or the dependence on external, proprietary models like GPT-4 for
judgment. It uniquely utilizes the LLMs being tested to compute the probability
discrepancy between the initial response and its revised versions. A higher
discrepancy for a given query between two LLMs indicates a relatively weaker
capability. Our findings reveal that ProbDiff achieves results on par with
those obtained from evaluations based on GPT-4, spanning a range of scenarios
that include natural language generation (NLG) tasks such as translation,
summarization, and our proposed Xiaohongshu blog writing task, and benchmarks
for LLM evaluation like AlignBench, MT-Bench, and AlpacaEval, across LLMs of
varying magnitudes.
