  A critical approach for efficiently deploying computationally demanding large
language models (LLMs) is Key-Value (KV) caching. The KV cache stores key-value
states of previously generated tokens, significantly reducing the need for
repetitive computations and thereby lowering latency in autoregressive
generation. However, the size of the KV cache grows linearly with sequence
length, posing challenges for applications requiring long context input and
extensive sequence generation. In this paper, we present a simple yet effective
approach, called MiniCache, to compress the KV cache across layers from a novel
depth perspective, significantly reducing the memory footprint for LLM
inference. Our approach is based on the observation that KV cache states
exhibit high similarity between the adjacent layers in the middle-to-deep
portion of LLMs. To facilitate merging, we propose disentangling the states
into the magnitude and direction components, interpolating the directions of
the state vectors while preserving their lengths unchanged. Furthermore, we
introduce a token retention strategy to keep highly distinct state pairs
unmerged, thus preserving the information with minimal additional storage
overhead. Our MiniCache is training-free and general, complementing existing KV
cache compression strategies, such as quantization and sparsity. We conduct a
comprehensive evaluation of MiniCache utilizing various models including
LLaMA-2, LLaMA-3, Phi-3, Mistral, and Mixtral across multiple benchmarks,
demonstrating its exceptional performance in achieving superior compression
ratios and high throughput. On the ShareGPT dataset, LLaMA-2-7B with 4-bit
MiniCache achieves a remarkable compression ratio of up to 5.02x, enhances
inference throughput by approximately 5x, and reduces the memory footprint by
41% compared to the FP16 full cache baseline, all while maintaining
near-lossless performance.
