  Generative Large Language Models (LLMs) have demonstrated remarkable results
for a wide range of tasks. However, deploying these models for inference has
been a significant challenge due to their unprecedented resource requirements.
This has forced existing deployment frameworks to use multi-GPU inference
pipelines, which are often complex and costly, or to use smaller and less
performant models. In this work, we demonstrate that the main bottleneck for
generative inference with LLMs is memory bandwidth, rather than compute,
specifically for single batch inference. While quantization has emerged as a
promising solution by representing weights with reduced precision, previous
efforts have often resulted in notable performance degradation. To address
this, we introduce SqueezeLLM, a post-training quantization framework that not
only enables lossless compression to ultra-low precisions of up to 3-bit, but
also achieves higher quantization performance under the same memory constraint.
Our framework incorporates two novel ideas: (i) sensitivity-based non-uniform
quantization, which searches for the optimal bit precision assignment based on
second-order information; and (ii) the Dense-and-Sparse decomposition that
stores outliers and sensitive weight values in an efficient sparse format. When
applied to the LLaMA models, our 3-bit quantization significantly reduces the
perplexity gap from the FP16 baseline by up to 2.1x as compared to the
state-of-the-art methods with the same memory requirement. Furthermore, when
deployed on an A6000 GPU, our quantized models achieve up to 2.3x speedup
compared to the baseline. Our code is available at
https://github.com/SqueezeAILab/SqueezeLLM.
