  Emerging real-time multi-model ML (RTMM) workloads such as AR/VR and drone
control involve dynamic behaviors in various granularity; task, model, and
layers within a model. Such dynamic behaviors introduce new challenges to the
system software in an ML system since the overall system load is not completely
predictable, unlike traditional ML workloads. In addition, RTMM workloads
require real-time processing, involve highly heterogeneous models, and target
resource-constrained devices. Under such circumstances, developing an effective
scheduler gains more importance to better utilize underlying hardware
considering the unique characteristics of RTMM workloads. Therefore, we propose
a new scheduler, DREAM, which effectively handles various dynamicity in RTMM
workloads targeting multi-accelerator systems. DREAM quantifies the unique
requirements for RTMM workloads and utilizes the quantified scores to drive
scheduling decisions, considering the current system load and other inference
jobs on different models and input frames. DREAM utilizes tunable parameters
that provide fast and effective adaptivity to dynamic workload changes. In our
evaluation of five scenarios of RTMM workload, DREAM reduces the overall
UXCost, which is an equivalent metric of the energy-delay product (EDP) for
RTMM defined in the paper, by 32.2% and 50.0% in the geometric mean (up to
80.8% and 97.6%) compared to state-of-the-art baselines, which shows the
efficacy of our scheduling methodology.
