  To overcome the overparameterized problem in Pre-trained Language Models
(PLMs), pruning is widely used as a simple and straightforward compression
method by directly removing unimportant weights. Previous first-order methods
successfully compress PLMs to extremely high sparsity with little performance
drop. These methods, such as movement pruning, use first-order information to
prune PLMs while fine-tuning the remaining weights. In this work, we argue
fine-tuning is redundant for first-order pruning, since first-order pruning is
sufficient to converge PLMs to downstream tasks without fine-tuning. Under this
motivation, we propose Static Model Pruning (SMP), which only uses first-order
pruning to adapt PLMs to downstream tasks while achieving the target sparsity
level. In addition, we also design a new masking function and training
objective to further improve SMP. Extensive experiments at various sparsity
levels show SMP has significant improvements over first-order and zero-order
methods. Unlike previous first-order methods, SMP is also applicable to low
sparsity and outperforms zero-order methods. Meanwhile, SMP is more parameter
efficient than other methods due to it does not require fine-tuning.
