  $ $Large language models (LLMs) have become an increasingly important
workload for cloud providers catering to both enterprise and consumer
applications. LLM inference requests from these applications have end-to-end
latency SLOs that must be adhered to in production settings. However, existing
LLM serving systems focus on optimization objectives such as request serving
throughput or request execution latency rather than the end-to-end latency
SLOs. Achieving end-to-end SLOs for latency-sensitive requests is challenging
due to head-of-line (HOL) blocking in the request queue, which results from
bursty arrival rates and insufficient resources.
  To address the above challenge, we propose QLM, a multi-model queue
management framework for LLM serving. QLM uses stochastic programming to
orchestrate the actions of multiple LLM Serving Operations (LSOs) to reduce HOL
blocking and maximize SLO attainment. Specifically, QLM uses the following
LSOs: model swapping, request eviction, GPU-CPU state swapping, load balancing,
and warm model start. Evaluation on heterogeneous GPU devices and models with
real-world LLM serving dataset shows that QLM improves SLO attainment by 40-90%
and throughput by 20-400% while maintaining or improving device utilization
compared to other state-of-the-art LLM serving systems.
