  There is an increasing interest in developing LLMs for medical diagnosis to
improve diagnosis efficiency. Despite their alluring technological potential,
there is no unified and comprehensive evaluation criterion, leading to the
inability to evaluate the quality and potential risks of medical LLMs, further
hindering the application of LLMs in medical treatment scenarios. Besides,
current evaluations heavily rely on labor-intensive interactions with LLMs to
obtain diagnostic dialogues and human evaluation on the quality of diagnosis
dialogue. To tackle the lack of unified and comprehensive evaluation criterion,
we first initially establish an evaluation criterion, termed LLM-specific
Mini-CEX to assess the diagnostic capabilities of LLMs effectively, based on
original Mini-CEX. To address the labor-intensive interaction problem, we
develop a patient simulator to engage in automatic conversations with LLMs, and
utilize ChatGPT for evaluating diagnosis dialogues automatically. Experimental
results show that the LLM-specific Mini-CEX is adequate and necessary to
evaluate medical diagnosis dialogue. Besides, ChatGPT can replace manual
evaluation on the metrics of humanistic qualities and provides reproducible and
automated comparisons between different LLMs.
