  Large language models (LLMs) iteratively generate text token by token, with
memory usage increasing with the length of generated token sequences. The
unpredictability of generation lengths makes it difficult to estimate the time
and memory needed to process requests, posing a challenge for effective request
scheduling. Conventional sequence-level scheduling (SLS) serves requests in a
first-come first-served (FCFS) manner with static batching where requests with
short generation lengths are delayed until those with long ones have finished
generation, which hurts computational efficiency. Besides, to avoid
out-of-memory (OOM) errors, SLS batches requests with a small batch size, which
limits throughput. Recently proposed iteration-level scheduling (ILS) enhances
computational efficiency with continuous batching to return completed requests
timely and dynamically add new requests for processing. However, many ILS
schedulers limit the number of parallel-processing requests to avoid OOM errors
while achieving a fast inference speed, which compromises throughput. Moreover,
existing SLS and ILS schedulers fail to balance the workload across multiple
deployed LLM instances. To tackle these challenges, we propose slice-level
scheduling (SCLS). By splitting the predefined maximal generation length limit
into slices and serving batches slice by slice, it provides a precise range of
serving time and memory usage for batched requests, laying the foundation for
effective scheduling. Experiments confirm that compared with SLS and ILS
schedulers, SCLS can improve throughput by up to 315.8% and greatly mitigate
load imbalance with proposed batching and offloading algorithms.
