  Evaluating the quality of generated text is a challenging task in NLP, due to
the inherent complexity and diversity of text. Recently, large language models
(LLMs) have garnered significant attention due to their impressive performance
in various tasks. Therefore, we present this paper to investigate the
effectiveness of LLMs, especially ChatGPT, and explore ways to optimize their
use in assessing text quality. We compared three kinds of reference-free
evaluation methods. The experimental results prove that ChatGPT is capable of
evaluating text quality effectively from various perspectives without reference
and demonstrates superior performance than most existing automatic metrics. In
particular, the Explicit Score, which utilizes ChatGPT to generate a numeric
score measuring text quality, is the most effective and reliable method among
the three exploited approaches. However, directly comparing the quality of two
texts may lead to suboptimal results. We believe this paper will provide
valuable insights for evaluating text quality with LLMs and have released the
used data.
