  Automated generation of clinically accurate radiology reports can improve
patient care. Previous report generation methods that rely on image captioning
models often generate incoherent and incorrect text due to their lack of
relevant domain knowledge, while retrieval-based attempts frequently retrieve
reports that are irrelevant to the input image. In this work, we propose
Contrastive X-Ray REport Match (X-REM), a novel retrieval-based radiology
report generation module that uses an image-text matching score to measure the
similarity of a chest X-ray image and radiology report for report retrieval. We
observe that computing the image-text matching score with a language-image
model can effectively capture the fine-grained interaction between image and
text that is often lost when using cosine similarity. X-REM outperforms
multiple prior radiology report generation modules in terms of both natural
language and clinical metrics. Human evaluation of the generated reports
suggests that X-REM increased the number of zero-error reports and decreased
the average error severity compared to the baseline retrieval approach. Our
code is available at: https://github.com/rajpurkarlab/X-REM
