  Large language models (LLMs) can now handle longer sequences of tokens,
enabling complex tasks like book understanding and generating lengthy novels.
However, the key-value (KV) cache required for LLMs consumes substantial memory
as context length increasing, becoming the bottleneck for deployment. In this
paper, we present a strategy called SKVQ, which stands for sliding-window KV
cache quantization, to address the issue of extremely low bitwidth KV cache
quantization. To achieve this, SKVQ rearranges the channels of the KV cache in
order to improve the similarity of channels in quantization groups, and applies
clipped dynamic quantization at the group level. Additionally, SKVQ ensures
that the most recent window tokens in the KV cache are preserved with high
precision. This helps maintain the accuracy of a small but important portion of
the KV cache.SKVQ achieves high compression ratios while maintaining accuracy.
Our evaluation on LLMs demonstrates that SKVQ surpasses previous quantization
approaches, allowing for quantization of the KV cache to 2-bit keys and 1.5-bit
values with minimal loss of accuracy. With SKVQ, it is possible to process
context lengths of up to 1M on an 80GB memory GPU for a 7b model and up to 7
times faster decoding.
