  The integration of Artificial Intelligence (AI), especially Large Language
Models (LLMs), into the clinical diagnosis process offers significant potential
to improve the efficiency and accessibility of medical care. While LLMs have
shown some promise in the medical domain, their application in clinical
diagnosis remains underexplored, especially in real-world clinical practice,
where highly sophisticated, patient-specific decisions need to be made. Current
evaluations of LLMs in this field are often narrow in scope, focusing on
specific diseases or specialties and employing simplified diagnostic tasks. To
bridge this gap, we introduce CliBench, a novel benchmark developed from the
MIMIC IV dataset, offering a comprehensive and realistic assessment of LLMs'
capabilities in clinical diagnosis. This benchmark not only covers diagnoses
from a diverse range of medical cases across various specialties but also
incorporates tasks of clinical significance: treatment procedure
identification, lab test ordering and medication prescriptions. Supported by
structured output ontologies, CliBench enables a precise and multi-granular
evaluation, offering an in-depth understanding of LLM's capability on diverse
clinical tasks of desired granularity. We conduct a zero-shot evaluation of
leading LLMs to assess their proficiency in clinical decision-making. Our
preliminary results shed light on the potential and limitations of current LLMs
in clinical settings, providing valuable insights for future advancements in
LLM-powered healthcare.
