  The existence of redundancy in Convolutional Neural Networks (CNNs) enables
us to remove some filters/channels with acceptable performance drops. However,
the training objective of CNNs usually tends to minimize an accuracy-related
loss function without any attention paid to the redundancy, making the
redundancy distribute randomly on all the filters, such that removing any of
them may trigger information loss and accuracy drop, necessitating a following
finetuning step for recovery. In this paper, we propose to manipulate the
redundancy during training to facilitate network pruning. To this end, we
propose a novel Centripetal SGD (C-SGD) to make some filters identical,
resulting in ideal redundancy patterns, as such filters become purely redundant
due to their duplicates; hence removing them does not harm the network. As
shown on CIFAR and ImageNet, C-SGD delivers better performance because the
redundancy is better organized, compared to the existing methods. The
efficiency also characterizes C-SGD because it is as fast as regular SGD,
requires no finetuning, and can be conducted simultaneously on all the layers
even in very deep CNNs. Besides, C-SGD can improve the accuracy of CNNs by
first training a model with the same architecture but wider layers then
squeezing it into the original width.
