  The demand for large language model (LLM) inference is gradually dominating
the artificial intelligence workloads. Therefore, there is an urgent need for
cost-efficient inference serving. Existing work focuses on single-worker
optimization and lacks consideration of cluster-level management for both
inference queries and computing resources. However, placing requests and
managing resources without considering the query features easily causes SLO
violations or resource underutilization. Providers are forced to allocate extra
computing resources to guarantee user experience, leading to additional serving
costs. In this paper we introduce Aladdin, a scheduler that co-adaptively
places queries and scales computing resources with SLO awareness. For a stream
of inference queries, Aladdin first predicts minimal computing resources and
the corresponding serving workers' configuration required to fulfill the SLOs
for all queries. Then, it places the queries to each serving worker according
to the prefill and decode latency models of batched LLM inference to maximize
each worker's utilization. Results show that Aladdin reduces the serving cost
of a single model by up to 71% for the same SLO level compared with the
baselines, which can be millions of dollars per year.
