  The demand for deploying large language models(LLMs) on mobile devices
continues to increase, driven by escalating data security concerns and cloud
costs. However, network bandwidth and memory limitations pose challenges for
deploying billion-level models on mobile devices. In this study, we investigate
the outputs of different layers across various scales of LLMs and found that
the outputs of most layers exhibit significant similarity. Moreover, this
similarity becomes more pronounced as the model size increases, indicating
substantial redundancy in the depth direction of the LLMs. Based on this
observation, we propose an efficient model volume compression strategy, termed
FoldGPT, which combines block removal and block parameter sharing.This strategy
consists of three parts: (1) Based on the learnable gating parameters, we
determine the block importance ranking while modeling the coupling effect
between blocks. Then we delete some redundant layers based on the given removal
rate. (2) For the retained blocks, we apply a specially designed group
parameter sharing strategy, where blocks within the same group share identical
weights, significantly compressing the number of parameters and slightly
reducing latency overhead. (3) After sharing these Blocks, we "cure" the
mismatch caused by sparsity with a minor amount of fine-tuning and introduce a
tail-layer distillation strategy to improve the performance. Experiments
demonstrate that FoldGPT outperforms previous state-of-the-art(SOTA) methods in
efficient model compression, demonstrating the feasibility of achieving model
lightweighting through straightforward block removal and parameter sharing.
