  Fine-tuning large language models (LLMs) for downstream tasks can greatly
improve model quality, however serving many different fine-tuned LLMs
concurrently for users in multi-tenant environments is challenging. Dedicating
GPU memory for each model is prohibitively expensive and naively swapping large
model weights in and out of GPU memory is slow. Our key insight is that
fine-tuned models can be quickly swapped in and out of GPU memory by extracting
and compressing the delta between each model and its pre-trained base model. We
propose DeltaZip, an LLM serving system that efficiently serves multiple
full-parameter fine-tuned models concurrently by aggressively compressing model
deltas by a factor of $6\times$ to $8\times$ while maintaining high model
quality. DeltaZip increases serving throughput by $1.5\times$ to $3\times$ and
improves SLO attainment compared to a vanilla HuggingFace serving system.
