  Large language models (LLMs) have demonstrated remarkable performance, and
organizations are racing to serve LLMs of varying sizes as endpoints for
use-cases like chat, programming and search. However, efficiently serving
multiple LLMs poses significant challenges for existing approaches due to
varying popularity of LLMs. In the paper, we present MuxServe, a flexible
spatial-temporal multiplexing system for efficient multiple LLM serving. The
key insight behind is to colocate LLMs considering their popularity to
multiplex memory resources, and leverage the characteristics of prefill and
decoding phases to separate and flexibly colocate them to multiplex computation
resources. MuxServe formally formulates the multiplexing problem, and proposes
a novel placement algorithm and adaptive batch scheduling strategy to identify
optimal colocations and maximize utilization. MuxServe designs a unified
resource manager to enable flexible and efficient multiplexing. Evaluation
results show that MuxServe can achieves up to $1.8\times$ higher throughput or
processes $2.9\times$ more requests within $99\%$ SLO attainment. The code is
available at: \url{https://github.com/hao-ai-lab/MuxServe}.
