  The emergence of various medical large language models (LLMs) in the medical
domain has highlighted the need for unified evaluation standards, as manual
evaluation of LLMs proves to be time-consuming and labor-intensive. To address
this issue, we introduce MedBench, a comprehensive benchmark for the Chinese
medical domain, comprising 40,041 questions sourced from authentic examination
exercises and medical reports of diverse branches of medicine. In particular,
this benchmark is composed of four key components: the Chinese Medical
Licensing Examination, the Resident Standardization Training Examination, the
Doctor In-Charge Qualification Examination, and real-world clinic cases
encompassing examinations, diagnoses, and treatments. MedBench replicates the
educational progression and clinical practice experiences of doctors in
Mainland China, thereby establishing itself as a credible benchmark for
assessing the mastery of knowledge and reasoning abilities in medical language
learning models. We perform extensive experiments and conduct an in-depth
analysis from diverse perspectives, which culminate in the following findings:
(1) Chinese medical LLMs underperform on this benchmark, highlighting the need
for significant advances in clinical knowledge and diagnostic precision. (2)
Several general-domain LLMs surprisingly possess considerable medical
knowledge. These findings elucidate both the capabilities and limitations of
LLMs within the context of MedBench, with the ultimate goal of aiding the
medical research community.
