  We present an interpretable neural network for predicting an important
clinical outcome (1-year mortality) from multi-modal Electronic Health Record
(EHR) data. Our approach builds on prior multi-modal machine learning models by
now enabling visualization of how individual factors contribute to the overall
outcome risk, assuming other factors remain constant, which was previously
impossible.
  We demonstrate the value of this approach using a large multi-modal clinical
dataset including both EHR data and 31,278 echocardiographic videos of the
heart from 26,793 patients. We generated separate models for (i) clinical data
only (CD) (e.g. age, sex, diagnoses and laboratory values), (ii) numeric
variables derived from the videos, which we call echocardiography-derived
measures (EDM), and (iii) CD+EDM+raw videos (pixel data). The interpretable
multi-modal model maintained performance compared to non-interpretable models
(Random Forest, XGBoost), and also performed significantly better than a model
using a single modality (average AUC=0.82). Clinically relevant insights and
multi-modal variable importance rankings were also facilitated by the new
model, which have previously been impossible.
