  The evaluation of natural language generation (NLG) tasks is a significant
and longstanding research issue. With the recent emergence of powerful large
language models (LLMs), some studies have turned to LLM-based automatic
evaluation methods, which demonstrate great potential to become a new
evaluation paradigm following traditional string-based and model-based metrics.
However, despite the improved performance of existing methods, they still
possess some deficiencies, such as dependency on references and limited
evaluation flexibility. Therefore, in this paper, we meticulously construct a
large-scale NLG evaluation corpus NLG-Eval with human and GPT-4 annotations to
alleviate the lack of relevant data in this field. Furthermore, we propose
Themis, an LLM dedicated to NLG evaluation, which has been trained with our
designed multi-perspective consistency and rating-oriented preference alignment
methods. Themis can conduct flexible and interpretable evaluations without
references, and it exhibits superior evaluation performance on various NLG
tasks, simultaneously generalizing well to unseen tasks and surpassing other
evaluation models, including GPT-4.
