  Post-training quantization (PTQ) techniques applied to weights, activations,
and the KV cache greatly reduce memory usage, latency, and power consumption of
Large Language Models (LLMs), but may lead to large quantization errors when
outliers are present. Recent findings suggest that rotating activation or
weight matrices helps remove outliers and benefits quantization. In this work,
we identify a collection of applicable rotation parameterizations that lead to
identical outputs in full-precision Transformer architectures, and find that
some random rotations lead to much better quantization than others, with an up
to 13 points difference in downstream zero-shot reasoning performance. As a
result, we propose SpinQuant that optimizes (or learns) the rotation matrices
with Cayley optimization on a small validation set. With 4-bit quantization of
weight, activation, and KV-cache, SpinQuant narrows the accuracy gap on
zero-shot reasoning tasks with full precision to merely 2.9 points on the
LLaMA-2 7B model, surpassing LLM-QAT by 19.1 points and SmoothQuant by 25.0
points. SpinQuant also outperforms concurrent work QuaRot, which applies random
rotations to remove outliers. In particular, for LLaMA-2 7B/LLaMA-3 8B models
that are hard to quantize, SpinQuant reduces the gap to full precision by
30.2%/34.1% relative to QuaRot.
