  Large Language Models (LLMs) provide a possibility to make a great
breakthrough in medicine. The establishment of a standardized medical benchmark
becomes a fundamental cornerstone to measure progression. However, medical
environments in different regions have their local characteristics, e.g., the
ubiquity and significance of traditional Chinese medicine within China.
Therefore, merely translating English-based medical evaluation may result in
\textit{contextual incongruities} to a local region. To solve the issue, we
propose a localized medical benchmark called CMB, a Comprehensive Medical
Benchmark in Chinese, designed and rooted entirely within the native Chinese
linguistic and cultural framework. While traditional Chinese medicine is
integral to this evaluation, it does not constitute its entirety. Using this
benchmark, we have evaluated several prominent large-scale LLMs, including
ChatGPT, GPT-4, dedicated Chinese LLMs, and LLMs specialized in the medical
domain. We hope this benchmark provide first-hand experience in existing LLMs
for medicine and also facilitate the widespread adoption and enhancement of
medical LLMs within China. Our data and code are publicly available at
https://github.com/FreedomIntelligence/CMB.
