  Large-scale language models (LLMs) have demonstrated impressive performance,
but their deployment presents challenges due to their significant memory usage.
This issue can be alleviated through quantization. In this paper, we identify
that the challenge in quantizing activations in LLMs arises from varying ranges
across channels, rather than solely the presence of outliers. To address this
challenge, we introduce a quantization method called RPTQ, which utilizes a
reorder-based approach. By rearranging the channels and quantizing them in
clusters, RPTQ effectively mitigates the impact of range differences between
channels. To minimize the overhead of the reorder operation, we fuse it into
the layer norm operation and weights in linear layers. In our experiments, RPTQ
achieved a significant breakthrough by utilizing 3-bit activation in LLMs for
the first time, resulting in a substantial reduction in memory usage. For
instance, quantizing OPT-175b can lead to a memory consumption reduction of up
to 80%.
