  Deep Neural Networks have been used in a wide variety of applications with
significant success. However, their highly complex nature owing to comprising
millions of parameters has lead to problems during deployment in pipelines with
low latency requirements. As a result, it is more desirable to obtain
lightweight neural networks which have the same performance during inference
time. In this work, we propose a weight based pruning approach in which the
weights are pruned gradually based on their momentum of the previous
iterations. Each layer of the neural network is assigned an importance value
based on their relative sparsity, followed by the magnitude of the weight in
the previous iterations. We evaluate our approach on networks such as AlexNet,
VGG16 and ResNet50 with image classification datasets such as CIFAR-10 and
CIFAR-100. We found that the results outperformed the previous approaches with
respect to accuracy and compression ratio. Our method is able to obtain a
compression of 15% for the same degradation in accuracy on both the datasets.
