  Structured pruning of modern large language models (LLMs) has emerged as a
way of decreasing their high computational needs. Width pruning reduces the
size of projection weight matrices (e.g., by removing attention heads) while
maintaining the number of layers. Depth pruning, in contrast, removes entire
layers or blocks, while keeping the size of the remaining weights unchanged.
Most current research focuses on either width-only or a blend of width and
depth pruning, with little comparative analysis between the two units (width
vs. depth) concerning their impact on LLM inference efficiency. In this work,
we show that simple depth pruning can effectively compress LLMs while achieving
comparable or superior performance to recent width pruning studies. Our pruning
method boosts inference speeds, especially under memory-constrained conditions
that require limited batch sizes for running LLMs, where width pruning is
ineffective. In retraining pruned models for quality recovery, continued
pretraining on a large corpus markedly outperforms LoRA-based tuning,
particularly at severe pruning ratios. We hope this work can help build compact
yet capable LLMs. Code and models can be found at:
https://github.com/Nota-NetsPresso/shortened-llm
