  We present a simple variable quantization approach that quantizes different
layers of a large language model (LLM) at different bit levels. Specifically,
we quantize the most important layers to higher bit precision and less
important layers to lower bits to achieve floating point quantization levels.
We propose two effective strategies to measure the importance of layers within
LLMs: the first measures the importance of a layer based on how different its
output embeddings are from the input embeddings (the higher the better); the
second estimates the importance of a layer using the number of layer weights
that are much larger than average (the smaller the better). We show that
quantizing different layers at varying bits according to our importance scores
results in minimal performance drop with a far more compressed model size.
Finally, we present several practical key takeaways from our variable
layer-wise quantization experiments: (a) LLM performance under variable
quantization remains close to the original model until 25-50% of layers are
moved in lower quantization using our proposed ordering but only until 5-10% if
moved using no specific ordering; (b) Quantizing LLMs to lower bits performs
substantially better than pruning unless extreme quantization (2-bit) is used;
and (c) Layer-wise quantization to lower bits works better in the case of
larger LLMs with more layers compared to smaller LLMs with fewer layers. The
code used to run the experiments is available at:
https://github.com/RazvanDu/LayerwiseQuant.
