  Efficiently serving large language models (LLMs) requires batching of many
requests to reduce the cost per request. Yet, with larger batch sizes and
longer context lengths, the key-value (KV) cache, which stores attention keys
and values to avoid re-computations, significantly increases memory demands and
becomes the new bottleneck in speed and memory usage. Additionally, the loading
of the KV cache causes the computational core to be idle, which limits the
inference speed. A straightforward and effective solution to reduce KV cache
size is quantization, which decreases the total bytes taken by KV cache.
However, there is a lack of in-depth studies that explore the element
distribution of KV cache to understand the hardness and limitation of KV cache
quantization. To fill the gap, we conducted a comprehensive study on the
element distribution in KV cache of popular LLMs. Our findings indicate that
the key cache should be quantized per-channel, i.e., group elements along the
channel dimension and quantize them together. In contrast, the value cache
should be quantized per-token. From this analysis, we developed a tuning-free
2bit KV cache quantization algorithm named KIVI. With hardware-friendly
implementation, KIVI can enable Llama, Falcon, and Mistral models to maintain
almost the same quality while using $\mathbf{2.6\times}$ less peak memory
(including model weight). This reduction in memory usage enables up to
$\mathbf{4\times}$ larger batch size, bringing $\mathbf{2.35\times \sim
3.47\times}$ throughput on real LLM inference workload. The source code is
available at https://github.com/jy-yuan/KIVI.
