  This paper introduces LLM-Streamline, a novel layer pruning approach for
large language models. It is based on the observation that different layers
have varying impacts on hidden states, enabling the identification of less
important layers. LLMStreamline comprises two parts: layer pruning, which
removes consecutive layers with the lowest importance based on target sparsity,
and layer replacement, where a lightweight network is trained to replace the
pruned layers to mitigate performance loss. Additionally, a new metric called
"stability" is proposed to address the limitations of accuracy in evaluating
model compression. Experiments show that LLM-Streamline surpasses previous
state-of-the-art pruning methods in both accuracy and stability.
