  This paper quantitatively reveals the state-of-the-art and
state-of-the-practice AI systems only achieve acceptable performance on the
stringent conditions that all categories of subjects are known, which we call
closed clinical settings, but fail to work in real-world clinical settings.
Compared to the diagnosis task in the closed setting, real-world clinical
settings pose severe challenges, and we must treat them differently. We build a
clinical AI benchmark named Clinical AIBench to set up real-world clinical
settings to facilitate researches. We propose an open, dynamic machine learning
framework and develop an AI system named OpenClinicalAI to diagnose diseases in
real-world clinical settings. The first versions of Clinical AIBench and
OpenClinicalAI target Alzheimer's disease. In the real-world clinical setting,
OpenClinicalAI significantly outperforms the state-of-the-art AI system. In
addition, OpenClinicalAI develops personalized diagnosis strategies to avoid
unnecessary testing and seamlessly collaborates with clinicians. It is
promising to be embedded in the current medical systems to improve medical
services.
