  Network pruning is widely used to lighten and accelerate neural network
models. Structured network pruning discards the whole neuron or filter, leading
to accuracy loss. In this work, we propose a novel concept of neuron merging
applicable to both fully connected layers and convolution layers, which
compensates for the information loss due to the pruned neurons/filters. Neuron
merging starts with decomposing the original weights into two matrices/tensors.
One of them becomes the new weights for the current layer, and the other is
what we name a scaling matrix, guiding the combination of neurons. If the
activation function is ReLU, the scaling matrix can be absorbed into the next
layer under certain conditions, compensating for the removed neurons. We also
propose a data-free and inexpensive method to decompose the weights by
utilizing the cosine similarity between neurons. Compared to the pruned model
with the same topology, our merged model better preserves the output feature
map of the original model; thus, it maintains the accuracy after pruning
without fine-tuning. We demonstrate the effectiveness of our approach over
network pruning for various model architectures and datasets. As an example,
for VGG-16 on CIFAR-10, we achieve an accuracy of 93.16% while reducing 64% of
total parameters, without any fine-tuning. The code can be found here:
https://github.com/friendshipkim/neuron-merging
