  Deep Neural Networks are highly over-parameterized and the size of the neural
networks can be reduced significantly after training without any decrease in
performance. One can clearly see this phenomenon in a wide range of
architectures trained for various problems. Weight/channel pruning,
distillation, quantization, matrix factorization are some of the main methods
one can use to remove the redundancy to come up with smaller and faster models.
  This work starts with a short informative chapter, where we motivate the
pruning idea and provide the necessary notation. In the second chapter, we
compare various saliency scores in the context of parameter pruning. Using the
insights obtained from this comparison and stating the problems it brings we
motivate why pruning units instead of the individual parameters might be a
better idea. We propose some set of definitions to quantify and analyze units
that don't learn and create any useful information. We propose an efficient way
for detecting dead units and use it to select which units to prune. We get 5x
model size reduction through unit-wise pruning on MNIST.
