  Artificial intelligence (AI) systems hold great promise to improve healthcare
over the next decades. Specifically, AI systems leveraging multiple data
sources and input modalities are poised to become a viable method to deliver
more accurate results and deployable pipelines across a wide range of
applications. In this work, we propose and evaluate a unified Holistic AI in
Medicine (HAIM) framework to facilitate the generation and testing of AI
systems that leverage multimodal inputs. Our approach uses generalizable data
pre-processing and machine learning modeling stages that can be readily adapted
for research and deployment in healthcare environments. We evaluate our HAIM
framework by training and characterizing 14,324 independent models based on
HAIM-MIMIC-MM, a multimodal clinical database (N=34,537 samples) containing
7,279 unique hospitalizations and 6,485 patients, spanning all possible input
combinations of 4 data modalities (i.e., tabular, time-series, text, and
images), 11 unique data sources and 12 predictive tasks. We show that this
framework can consistently and robustly produce models that outperform similar
single-source approaches across various healthcare demonstrations (by 6-33%),
including 10 distinct chest pathology diagnoses, along with length-of-stay and
48-hour mortality predictions. We also quantify the contribution of each
modality and data source using Shapley values, which demonstrates the
heterogeneity in data modality importance and the necessity of multimodal
inputs across different healthcare-relevant tasks. The generalizable properties
and flexibility of our Holistic AI in Medicine (HAIM) framework could offer a
promising pathway for future multimodal predictive systems in clinical and
operational healthcare settings.
