  Iterative Magnitude Pruning (IMP) is a network pruning method that repeats
the process of removing weights with the least magnitudes and retraining the
model. When visualizing the weight matrices of language models pruned by IMP,
previous research has shown that a structured pattern emerges, wherein the
resulting surviving weights tend to prominently cluster in a select few rows
and columns of the matrix. Though the need for further research in utilizing
these structured patterns for potential performance gains has previously been
indicated, it has yet to be thoroughly studied. We propose SPUR (Structured
Pattern pruning Using Regularization), a novel pruning mechanism that
preemptively induces structured patterns in compression by adding a
regularization term to the objective function in the IMP. Our results show that
SPUR can significantly preserve model performance under high sparsity settings
regardless of the language or the task. Our contributions are as follows: (i)
We propose SPUR, a network pruning mechanism that improves upon IMP regardless
of the language or the task. (ii) We are the first to empirically verify the
efficacy of "structured patterns" observed previously in pruning research.
(iii) SPUR is a resource-efficient mechanism in that it does not require
significant additional computations.
