  Automatically interpreting CT scans can ease the workload of radiologists.
However, this is challenging mainly due to the scarcity of adequate datasets
and reference standards for evaluation. This study aims to bridge this gap by
introducing a novel evaluation framework, named ``GPTRadScore''. This framework
assesses the capabilities of multi-modal LLMs, such as GPT-4 with Vision
(GPT-4V), Gemini Pro Vision, LLaVA-Med, and RadFM, in generating descriptions
for prospectively-identified findings. By employing a decomposition technique
based on GPT-4, GPTRadScore compares these generated descriptions with
gold-standard report sentences, analyzing their accuracy in terms of body part,
location, and type of finding. Evaluations demonstrated a high correlation with
clinician assessments and highlighted its potential over traditional metrics,
such as BLEU, METEOR, and ROUGE. Furthermore, to contribute to future studies,
we plan to release a benchmark dataset annotated by clinicians. Using
GPTRadScore, we found that while GPT-4V and Gemini Pro Vision fare better,
their performance revealed significant areas for improvement, primarily due to
limitations in the dataset used for training these models. To demonstrate this
potential, RadFM was fine-tuned and it resulted in significant accuracy
improvements: location accuracy rose from 3.41\% to 12.8\%, body part accuracy
from 29.12\% to 53\%, and type accuracy from 9.24\% to 30\%, thereby validating
our hypothesis.
