  GPT-4V's purported strong multimodal abilities raise interests in using it to
automate radiology report writing, but there lacks thorough evaluations. In
this work, we perform a systematic evaluation of GPT-4V in generating radiology
reports on two chest X-ray report datasets: MIMIC-CXR and IU X-Ray. We attempt
to directly generate reports using GPT-4V through different prompting
strategies and find that it fails terribly in both lexical metrics and clinical
efficacy metrics. To understand the low performance, we decompose the task into
two steps: 1) the medical image reasoning step of predicting medical condition
labels from images; and 2) the report synthesis step of generating reports from
(groundtruth) conditions. We show that GPT-4V's performance in image reasoning
is consistently low across different prompts. In fact, the distributions of
model-predicted labels remain constant regardless of which groundtruth
conditions are present on the image, suggesting that the model is not
interpreting chest X-rays meaningfully. Even when given groundtruth conditions
in report synthesis, its generated reports are less correct and less
natural-sounding than a finetuned LLaMA-2. Altogether, our findings cast doubt
on the viability of using GPT-4V in a radiology workflow.
