  Parameter-efficient finetuning (PEFT) is a widely used technique to adapt
large language models for different tasks. Service providers typically create
separate systems for users to perform PEFT model finetuning and inference
tasks. This is because existing systems cannot handle workloads that include a
mix of inference and PEFT finetuning requests. As a result, shared GPU
resources are underutilized, leading to inefficiencies. To address this
problem, we present FlexLLM, the first system that can serve inference and
parameter-efficient finetuning requests in the same iteration. Our system
leverages the complementary nature of these two tasks and utilizes shared GPU
resources to run them jointly, using a method called co-serving. To achieve
this, FlexLLM introduces a novel token-level finetuning mechanism, which breaks
down the finetuning computation of a sequence into smaller token-level
computations and uses dependent parallelization and graph pruning, two static
compilation optimizations, to minimize the memory overhead and latency for
co-serving. Compared to existing systems, FlexLLM's co-serving approach reduces
the activation GPU memory overhead by up to 8x, and the end-to-end GPU memory
requirement of finetuning by up to 36% while maintaining a low inference
latency and improving finetuning throughput. For example, under a heavy
inference workload, FlexLLM can still preserve more than 80% of the peak
finetuning throughput, whereas existing systems cannot make any progress with
finetuning. The source code of FlexLLM is publicly available at
https://github.com/flexflow/FlexFlow.
