  In this paper, we introduce a new post-training compression paradigm for
Large Language Models (LLMs) to facilitate their wider adoption. We delve into
LLM weight low-rank factorization, and find that the challenges of this task
stem from the outlier phenomenon in the LLM activations and the sensitivity
difference among various kinds of layers. To address these issues, we propose a
training-free approach called Activation-aware Singular Value Decomposition
(ASVD). Specifically, ASVD manages activation outliers by scaling the weight
matrix based on the activation distribution, thereby enhancing decomposition
accuracy. Additionally, we propose an efficient iterative calibration process
to optimize layer-specific decomposition by addressing the varying sensitivity
of different LLM layers. ASVD can compress a network by 10-20%, without
compromising the performance of LLMs. Based on the success of the low-rank
decomposition of projection matrices in the self-attention module, we further
introduce ASVD to compress the KV cache. By reducing the channel dimension of
KV activations, memory requirements for KV cache can be largely reduced. Thanks
to the 50-75% reduction in the rank of the KV projection matrices, ASVD can
further achieve 50% KV cache reductions without performance drop in a
training-free manner.
