  Recently, there has been a growing trend of utilizing Large Language Model
(LLM) to evaluate the quality of other LLMs. Many studies have employed
proprietary close-source models, especially GPT-4, as the evaluator.
Alternatively, other works have fine-tuned judge models based on open-source
LLMs as the evaluator. While the fine-tuned judge models are claimed to achieve
comparable evaluation capability with GPT-4, in this study, we conduct an
empirical study of judge models. Our findings indicate that although the
fine-tuned judge models achieve high performance on in-domain test sets, even
surpassing GPT-4, they underperform GPT-4 across several dimensions, including
generalizability, fairness, aspect-specific evaluation, and scalability. We
also reveal that the fine-tuned judge model inherently operates as a
task-specific classifier, consequently imposing the limitations. Finally, we
propose an effective indicator to measure the reliability of fine-tuned judges,
with the aim of maximizing their utility in LLM evaluation.
