  Real-time multi-task multi-model (MTMM) workloads, a new form of deep
learning inference workloads, are emerging for applications areas like extended
reality (XR) to support metaverse use cases. These workloads combine user
interactivity with computationally complex machine learning (ML) activities.
Compared to standard ML applications, these ML workloads present unique
difficulties and constraints. Real-time MTMM workloads impose heterogeneity and
concurrency requirements on future ML systems and devices, necessitating the
development of new capabilities. This paper begins with a discussion of the
various characteristics of these real-time MTMM ML workloads and presents an
ontology for evaluating the performance of future ML hardware for XR systems.
Next, we present XRBENCH, a collection of MTMM ML tasks, models, and usage
scenarios that execute these models in three representative ways: cascaded,
concurrent, and cascaded-concurrent for XR use cases. Finally, we emphasize the
need for new metrics that capture the requirements properly. We hope that our
work will stimulate research and lead to the development of a new generation of
ML systems for XR use cases. XRBench is available as an open-source project:
https://github.com/XRBench
