  Given a pretrained encoder-based language model, how can we accurately
compress it without retraining? Retraining-free structured pruning algorithms
are crucial in pretrained language model compression due to their significantly
reduced pruning cost and capability to prune large language models. However,
existing retraining-free algorithms encounter severe accuracy degradation, as
they fail to handle pruning errors, especially at high compression rates. In
this paper, we propose K-prune (Knowledge-preserving pruning), an accurate
retraining-free structured pruning algorithm for pretrained encoder-based
language models. K-prune focuses on preserving the useful knowledge of the
pretrained model to minimize pruning errors through a carefully designed
iterative pruning process composed of knowledge measurement,
knowledge-preserving mask search, and knowledge-preserving weight-tuning. As a
result, K-prune shows significant accuracy improvements up to 58.02%p higher F1
score compared to existing retraining-free pruning algorithms under a high
compression rate of 80% on the SQuAD benchmark without any retraining process.
