  This paper presents ServerlessLLM, a distributed system designed to support
low-latency serverless inference for Large Language Models (LLMs). By
harnessing the substantial near-GPU storage and memory capacities of inference
servers, ServerlessLLM achieves effective local checkpoint storage, minimizing
the need for remote checkpoint downloads and ensuring efficient checkpoint
loading. The design of ServerlessLLM features three core contributions: (i)
\emph{fast multi-tier checkpoint loading}, featuring a new loading-optimized
checkpoint format and a multi-tier loading system, fully utilizing the
bandwidth of complex storage hierarchies on GPU servers; (ii) \emph{efficient
live migration of LLM inference}, which enables newly initiated inferences to
capitalize on local checkpoint storage while ensuring minimal user
interruption; and (iii) \emph{startup-time-optimized model scheduling}, which
assesses the locality statuses of checkpoints on each server and schedules the
model onto servers that minimize the time to start the inference. Comprehensive
evaluations, including microbenchmarks and real-world scenarios, demonstrate
that ServerlessLLM dramatically outperforms state-of-the-art serverless
systems, reducing latency by 10 - 200X across various LLM inference workloads.
