  Modern model serving engines infer prompts on large language models in
batches. While batch processing prompts leads to high inference throughput, it
delays responding to requests that do not fit in a batch, potentially starving
them. We propose that fair scheduling prompts for inference by time-sharing
GPUs cycles, instead of batch processing them, is key to preventing prompt
starvation and achieving responsive inference. However, time-shared prompt
scheduling incurs the overhead of frequently paging dynamic context needed to
infer a prompt back into GPU memory. Today, serving engines support paging
inference context between GPU memory and the host DRAM. The overhead of
transferring context from DRAM to GPU memory is high since it is lower-bounded
by the limited PCIe bandwidth. We overcome this challenge by offloading
inference context from a GPU to the memory of another GPU on the same server,
connected via inter-GPU interconnects that support magnitudes higher bandwidth
than PCIe. We achieve this by developing AQUA, a transparent and elastic GPU
memory management framework for responsive LLM inference. We evaluate AQUA by
hosting eight state-of-the-art large generative ML models of different
modalities (e.g., text, audio, vision) on a server with 8 cutting-edge Nvidia
A100 80G GPUs. Using representative inference workloads, we show that AQUA
improves the responsiveness of LLM inference by 4X compared to the
state-of-the-art and it improves LLM inference throughput over a single long
prompt by 6X.
