  Deep learning models have achieved expert-level performance in healthcare
with an exclusive focus on training accurate models. However, in many clinical
environments such as intensive care unit (ICU), real-time model serving is
equally if not more important than accuracy, because in ICU patient care is
simultaneously more urgent and more expensive. Clinical decisions and their
timeliness, therefore, directly affect both the patient outcome and the cost of
care. To make timely decisions, we argue the underlying serving system must be
latency-aware. To compound the challenge, health analytic applications often
require a combination of models instead of a single model, to better specialize
individual models for different targets, multi-modal data, different prediction
windows, and potentially personalized predictions. To address these challenges,
we propose HOLMES-an online model ensemble serving framework for healthcare
applications. HOLMES dynamically identifies the best performing set of models
to ensemble for highest accuracy, while also satisfying sub-second latency
constraints on end-to-end prediction. We demonstrate that HOLMES is able to
navigate the accuracy/latency tradeoff efficiently, compose the ensemble, and
serve the model ensemble pipeline, scaling to simultaneously streaming data
from 100 patients, each producing waveform data at 250~Hz. HOLMES outperforms
the conventional offline batch-processed inference for the same clinical task
in terms of accuracy and latency (by order of magnitude). HOLMES is tested on
risk prediction task on pediatric cardio ICU data with above 95% prediction
accuracy and sub-second latency on 64-bed simulation.
