  In recent years, automated radiology report generation has experienced
significant growth. This paper introduces MRScore, an automatic evaluation
metric tailored for radiology report generation by leveraging Large Language
Models (LLMs). Conventional NLG (natural language generation) metrics like BLEU
are inadequate for accurately assessing the generated radiology reports, as
systematically demonstrated by our observations within this paper. To address
this challenge, we collaborated with radiologists to develop a framework that
guides LLMs for radiology report evaluation, ensuring alignment with human
analysis. Our framework includes two key components: i) utilizing GPT to
generate large amounts of training data, i.e., reports with different
qualities, and ii) pairing GPT-generated reports as accepted and rejected
samples and training LLMs to produce MRScore as the model reward. Our
experiments demonstrate MRScore's higher correlation with human judgments and
superior performance in model selection compared to traditional metrics. Our
code and datasets will be available on GitHub.
