  While large language models (LLMs) excel in many domains, their complexity
and scale challenge deployment in resource-limited environments. Current
compression techniques, such as parameter pruning, often fail to effectively
utilize the knowledge from pruned parameters. To address these challenges, we
propose Manifold-Based Knowledge Alignment and Layer Merging Compression (MKA),
a novel approach that uses manifold learning and the Normalized Pairwise
Information Bottleneck (NPIB) measure to merge similar layers, reducing model
size while preserving essential performance. We evaluate MKA on multiple
benchmark datasets and various LLMs. Our findings show that MKA not only
preserves model performance but also achieves substantial compression ratios,
outperforming traditional pruning methods. Moreover, when coupled with
quantization, MKA delivers even greater compression. Specifically, on the MMLU
dataset using the Llama3-8B model, MKA achieves a compression ratio of 43.75%
with a minimal performance decrease of only 2.82\%. The proposed MKA method
offers a resource-efficient and performance-preserving model compression
technique for LLMs.
