  The acceleration of Large Language Models (LLMs) research has opened up new
possibilities for evaluating generated texts. They serve as scalable and
economical evaluators, but the question of how reliable these evaluators are
has emerged as a crucial research question. Prior research efforts in the
meta-evaluation of LLMs as judges limit the prompting of an LLM to a single use
to obtain a final evaluation decision. They then compute the agreement between
LLMs' outputs and human labels. This lacks interpretability in understanding
the evaluation capability of LLMs. In light of this challenge, we propose
Decompose and Aggregate, which breaks down the evaluation process into
different stages based on pedagogical practices. Our experiments illustrate
that it not only provides a more interpretable window for how well LLMs
evaluate, but also leads to improvements up to 39.6% for different LLMs on a
variety of meta-evaluation benchmarks.
