  Large language models such as ChatGPT are increasingly explored in medical
domains. However, the absence of standard guidelines for performance evaluation
has led to methodological inconsistencies. This study aims to summarize the
available evidence on evaluating ChatGPT's performance in medicine and provide
direction for future research. We searched ten medical literature databases on
June 15, 2023, using the keyword "ChatGPT". A total of 3520 articles were
identified, of which 60 were reviewed and summarized in this paper and 17 were
included in the meta-analysis. The analysis showed that ChatGPT displayed an
overall integrated accuracy of 56% (95% CI: 51%-60%, I2 = 87%) in addressing
medical queries. However, the studies varied in question resource,
question-asking process, and evaluation metrics. Moreover, many studies failed
to report methodological details, including the version of ChatGPT and whether
each question was used independently or repeatedly. Our findings revealed that
although ChatGPT demonstrated considerable potential for application in
healthcare, the heterogeneity of the studies and insufficient reporting may
affect the reliability of these results. Further well-designed studies with
comprehensive and transparent reporting are needed to evaluate ChatGPT's
performance in medicine.
