  Nowadays, large language models (LLMs) are published as a service and can be
accessed by various applications via APIs, also known as
language-model-as-a-service (LMaaS). Without knowing the generation length of
requests, existing serving systems serve requests in a first-come, first-served
(FCFS) manner with a fixed batch size, which leads to two problems that affect
batch serving efficiency. First, the generation lengths of requests in a batch
vary, and requests with short generation lengths must wait for requests with
long generation lengths to finish during the batch serving procedure. Second,
requests with longer generation lengths consume more memory during serving.
Without knowing the generation lengths of batched requests, the batch size is
always set small to avoid the out-of-memory (OOM) error, thus preventing the
GPU from being fully utilized. In this paper, we find that a significant number
of popular applications in the LMaaS scenario have a positive correlation
between the generation length and the length of raw user input. Based on this
observation, we propose Magnus, which can accurately predict the request
generation length with the user input length, application-level, and user-level
semantic features. Accordingly, Magnus can achieve high request throughput by
batching requests of similar generation lengths together with adaptive batch
sizes. Besides, Magnus can also schedule batches with the highest response
ratio next (HRRN) policy to reduce request response time. Experiments conducted
on our testbed show that Magnus improves request throughput by up to 234\% and
reduces response time by up to 89.7\% compared to baselines.
