  This paper addresses the critical challenge of managing Quality of Service
(QoS) in cloud services, focusing on the nuances of individual tenant
expectations and varying Service Level Indicators (SLIs). It introduces a novel
approach utilizing Deep Reinforcement Learning for tenant-specific QoS
management in multi-tenant, multi-accelerator cloud environments. The chosen
SLI, deadline hit rate, allows clients to tailor QoS for each service request.
A novel online scheduling algorithm for Deep Neural Networks in
multi-accelerator systems is proposed, with a focus on guaranteeing
tenant-wise, model-specific QoS levels while considering real-time constraints.
