  Recently, considerable efforts have been directed towards compressing Large
Language Models (LLMs), which showcase groundbreaking capabilities across
diverse applications but entail significant deployment costs due to their large
sizes. Meanwhile, much less attention has been given to mitigating the costs
associated with deploying multiple LLMs of varying sizes despite its practical
significance. Thus, this paper introduces \emph{any-precision LLM}, extending
the concept of any-precision DNN to LLMs. Addressing challenges in
any-precision LLM, we propose a lightweight method for any-precision
quantization of LLMs, leveraging a post-training quantization framework, and
develop a specialized software engine for its efficient serving. As a result,
our solution significantly reduces the high costs of deploying multiple,
different-sized LLMs by overlaying LLMs quantized to varying bit-widths, such
as 3, 4, ..., $n$ bits, into a memory footprint comparable to a single $n$-bit
LLM. All the supported LLMs with varying bit-widths demonstrate
state-of-the-art model quality and inference throughput, proving itself to be a
compelling option for deployment of multiple, different-sized LLMs. Our code is
open-sourced and available online.
