  Critique, as a natural language description for assessing the quality of
model-generated content, has played a vital role in the training, evaluation,
and refinement of LLMs. However, a systematic method to evaluate the quality of
critique is lacking. In this paper, we pioneer the critique of critique, termed
MetaCritique, which builds specific quantification criteria. To achieve a
reliable evaluation outcome, we propose Atomic Information Units (AIUs), which
describe the critique in a more fine-grained manner. MetaCritique aggregates
each AIU's judgment for the overall score. Moreover, MetaCritique delivers a
natural language rationale for the intricate reasoning within each judgment.
Lastly, we construct a meta-evaluation dataset covering 4 tasks across 16
public datasets involving human-written and LLM-generated critiques.
Experiments demonstrate that MetaCritique can achieve near-human performance.
Our study can facilitate future research in LLM critiques based on our
following observations and released resources: (1) superior critiques judged by
MetaCritique can lead to better refinements, indicating that it can potentially
enhance the alignment of existing LLMs; (2) the leaderboard of critique models
reveals that open-source critique models commonly suffer from factuality
issues; (3) relevant code and data are publicly available at
https://github.com/GAIR-NLP/MetaCritique to support deeper exploration; (4) an
API at PyPI with the usage documentation in Appendix C allows users to assess
the critique conveniently.
