  Network Pruning is a promising way to address the huge computing resource
demands of the deployment and inference of Large Language Models (LLMs).
Retraining-free is important for LLMs' pruning methods. However, almost all of
the existing retraining-free pruning approaches for LLMs focus on unstructured
pruning, which requires specific hardware support for acceleration. In this
paper, we propose a novel retraining-free structured pruning framework for
LLMs, named FLAP (FLuctuation-based Adaptive Structured Pruning). It is
hardware-friendly by effectively reducing storage and enhancing inference
speed. For effective structured pruning of LLMs, we highlight three critical
elements that demand the utmost attention: formulating structured importance
metrics, adaptively searching the global compressed model, and implementing
compensation mechanisms to mitigate performance loss. First, FLAP determines
whether the output feature map is easily recoverable when a column of weight is
removed, based on the fluctuation pruning metric. Then it standardizes the
importance scores to adaptively determine the global compressed model
structure. At last, FLAP adds additional bias terms to recover the output
feature maps using the baseline values. We thoroughly evaluate our approach on
a variety of language benchmarks. Without any retraining, our method
significantly outperforms the state-of-the-art methods, including LLM-Pruner
and the extension of Wanda in structured pruning. The code is released at
https://github.com/CASIA-IVA-Lab/FLAP.
