  Large language models (LLMs) have demonstrated impressive abilities in
various domains while the inference cost is expensive. Many previous studies
exploit quantization methods to reduce LLM inference cost by reducing latency
and memory consumption. Applying 2-bit single-precision weight quantization
brings >3% accuracy loss, so the state-of-the-art methods use mixed-precision
methods for LLMs (e.g. Llama2-7b, etc.) to improve the accuracy. However,
challenges still exist: (1) Uneven distribution in weight matrix. (2) Large
speed degradation by adding sparse outliers. (3) Time-consuming dequantization
operations on GPUs. To tackle these challenges and enable fast and efficient
LLM inference on GPUs, we propose the following techniques in this paper. (1)
Intra-weight mixed-precision quantization. (2) Exclusive 2-bit sparse outlier
with minimum speed degradation. (3) Asynchronous dequantization. We conduct
extensive experiments on different model families (e.g. Llama3, etc.) and model
sizes. We achieve 2.91-bit for each weight considering all scales/zeros for
different models with negligible loss. As a result, with our 2/4/16
mixed-precision quantization for each weight matrix and asynchronous
dequantization during inference, our design achieves an end-to-end speedup for
Llama2-7b is 1.74x over the original model, and we reduce both runtime cost and
total cost by up to 2.53x and 2.29x with less GPU requirements.
