  The high computational and memory requirements of large language model (LLM)
inference make it feasible only with multiple high-end accelerators. Motivated
by the emerging demand for latency-insensitive tasks with batched processing,
this paper initiates the study of high-throughput LLM inference using limited
resources, such as a single commodity GPU. We present FlexGen, a
high-throughput generation engine for running LLMs with limited GPU memory.
FlexGen can be flexibly configured under various hardware resource constraints
by aggregating memory and computation from the GPU, CPU, and disk. By solving a
linear programming problem, it searches for efficient patterns to store and
access tensors. FlexGen further compresses the weights and the attention cache
to 4 bits with negligible accuracy loss. These techniques enable FlexGen to
have a larger space of batch size choices and thus significantly increase
maximum throughput. As a result, when running OPT-175B on a single 16GB GPU,
FlexGen achieves significantly higher throughput compared to state-of-the-art
offloading systems, reaching a generation throughput of 1 token/s for the first
time with an effective batch size of 144. On the HELM benchmark, FlexGen can
benchmark a 30B model with a 16GB GPU on 7 representative sub-scenarios in 21
hours. The code is available at https://github.com/FMInference/FlexGen
