  Large language models (LLMs) propel the prosperity of interactive AI
applications showcased by ChatGPT that demand timely response of inference
services. However, LLM inference is computation intensive and memory intensive,
and improper parameter configuration at LLM platforms may exacerbate the
inference time. In this paper, we analyze the impact of LLM output token
distribution on the inference queueing delay, where the max-token clipping and
the batched inference are considered. By formulating an M/G/1 model, we observe
that enforcing a maximum output token limit on a very small fraction of
inference requests can significantly reduce the queueing delay, and our model
facilitates the selection of the optimal limit. For the batch inference, we
model the service process as a bulk queue in which the batch processing time is
affected by the batch size and the maximum token size inside this batch
jointly. The queueing delays of the batching of all buffered requests (dynamic
batching), the batching of constant number of requests (fixed batching), and
the batching without intra-batch waiting (elastic batching) are derived.
Experimental results show that our mathematical models coincide with the
event-driven simulations well.
