  Many Neural Network Pruning approaches consist of several iterative training
and pruning steps, seemingly losing a significant amount of their performance
after pruning and then recovering it in the subsequent retraining phase. Recent
works of Renda et al. (2020) and Le & Hua (2021) demonstrate the significance
of the learning rate schedule during the retraining phase and propose specific
heuristics for choosing such a schedule for IMP (Han et al., 2015). We place
these findings in the context of the results of Li et al. (2020) regarding the
training of models within a fixed training budget and demonstrate that,
consequently, the retraining phase can be massively shortened using a simple
linear learning rate schedule. Improving on existing retraining approaches, we
additionally propose a method to adaptively select the initial value of the
linear schedule. Going a step further, we propose similarly imposing a budget
on the initial dense training phase and show that the resulting simple and
efficient method is capable of outperforming significantly more complex or
heavily parameterized state-of-the-art approaches that attempt to sparsify the
network during training. These findings not only advance our understanding of
the retraining phase, but more broadly question the belief that one should aim
to avoid the need for retraining and reduce the negative effects of 'hard'
pruning by incorporating the sparsification process into the standard training.
