  Quantizing large language models (LLMs) presents significant challenges,
primarily due to outlier activations that compromise the efficiency of low-bit
representation. Traditional approaches mainly focus on solving Normal
Outliers-activations with consistently high magnitudes across all tokens.
However, these techniques falter when dealing with Massive Outliers, which are
significantly higher in value and often cause substantial performance losses
during low-bit quantization. In this study, we propose DuQuant, an innovative
quantization strategy employing rotation and permutation transformations to
more effectively eliminate both types of outliers. Initially, DuQuant
constructs rotation matrices informed by specific outlier dimensions,
redistributing these outliers across adjacent channels within different
rotation blocks. Subsequently, a zigzag permutation is applied to ensure a
balanced distribution of outliers among blocks, minimizing block-wise variance.
An additional rotation further enhances the smoothness of the activation
landscape, thereby improving model performance. DuQuant streamlines the
quantization process and demonstrates superior outlier management, achieving
top-tier results in multiple tasks with various LLM architectures even under
4-bit weight-activation quantization. Our code is available at
https://github.com/Hsu1023/DuQuant.
