  We revisit the performance of the classic gradual magnitude pruning (GMP)
baseline for large language models, focusing on the classic BERT benchmark on
various popular tasks. Despite existing evidence in the literature that GMP
performs poorly, we show that a simple and general variant, which we call GMP*,
can match and sometimes outperform more complex state-of-the-art methods. Our
results provide a simple yet strong baseline for future work, highlight the
importance of parameter tuning for baselines, and even improve the performance
of the state-of-the-art second-order pruning method in this setting.
