  Biomedical language understanding benchmarks are the driving forces for
artificial intelligence applications with large language model (LLM) back-ends.
However, most current benchmarks: (a) are limited to English which makes it
challenging to replicate many of the successes in English for other languages,
or (b) focus on knowledge probing of LLMs and neglect to evaluate how LLMs
apply these knowledge to perform on a wide range of bio-medical tasks, or (c)
have become a publicly available corpus and are leaked to LLMs during
pre-training. To facilitate the research in medical LLMs, we re-build the
Chinese Biomedical Language Understanding Evaluation (CBLUE) benchmark into a
large scale prompt-tuning benchmark, PromptCBLUE. Our benchmark is a suitable
test-bed and an online platform for evaluating Chinese LLMs' multi-task
capabilities on a wide range bio-medical tasks including medical entity
recognition, medical text classification, medical natural language inference,
medical dialogue understanding and medical content/dialogue generation. To
establish evaluation on these tasks, we have experimented and report the
results with the current 9 Chinese LLMs fine-tuned with differtent fine-tuning
techniques.
