  Generative Artificial Intelligence (AI) can be used to automatically generate
medical reports based on transcripts of medical consultations. The aim is to
reduce the administrative burden that healthcare professionals face. The
accuracy of the generated reports needs to be established to ensure their
correctness and usefulness. There are several metrics for measuring the
accuracy of AI generated reports, but little work has been done towards the
application of these metrics in medical reporting. A comparative
experimentation of 10 accuracy metrics has been performed on AI generated
medical reports against their corresponding General Practitioner's (GP) medical
reports concerning Otitis consultations. The number of missing, incorrect, and
additional statements of the generated reports have been correlated with the
metric scores. In addition, we introduce and define a Composite Accuracy Score
which produces a single score for comparing the metrics within the field of
automated medical reporting. Findings show that based on the correlation study
and the Composite Accuracy Score, the ROUGE-L and Word Mover's Distance metrics
are the preferred metrics, which is not in line with previous work. These
findings help determine the accuracy of an AI generated medical report, which
aids the development of systems that generate medical reports for GPs to reduce
the administrative burden.
