  Curated datasets for healthcare are often limited due to the need of human
annotations from experts. In this paper, we present MedEval, a multi-level,
multi-task, and multi-domain medical benchmark to facilitate the development of
language models for healthcare. MedEval is comprehensive and consists of data
from several healthcare systems and spans 35 human body regions from 8
examination modalities. With 22,779 collected sentences and 21,228 reports, we
provide expert annotations at multiple levels, offering a granular potential
usage of the data and supporting a wide range of tasks. Moreover, we
systematically evaluated 10 generic and domain-specific language models under
zero-shot and finetuning settings, from domain-adapted baselines in healthcare
to general-purposed state-of-the-art large language models (e.g., ChatGPT). Our
evaluations reveal varying effectiveness of the two categories of language
models across different tasks, from which we notice the importance of
instruction tuning for few-shot usage of large language models. Our
investigation paves the way toward benchmarking language models for healthcare
and provides valuable insights into the strengths and limitations of adopting
large language models in medical domains, informing their practical
applications and future advancements.
