  Distributed LLM serving is costly and often underutilizes hardware
accelerators due to three key challenges: bubbles in pipeline-parallel
deployments caused by the bimodal latency of prompt and token processing, GPU
memory overprovisioning, and long recovery times in case of failures. In this
paper, we propose D\'ej\`aVu, a system to address all these challenges using a
versatile and efficient KV cache streaming library (D\'ej\`aVuLib). Using
D\'ej\`aVuLib, we propose and implement efficient prompt-token disaggregation
to reduce pipeline bubbles, microbatch swapping for efficient GPU memory
management, and state replication for fault-tolerance. We highlight the
efficacy of these solutions on a range of large models across cloud
deployments.
