  Network pruning is aimed at imposing sparsity in a neural network
architecture by increasing the portion of zero-valued weights for reducing its
size regarding energy-efficiency consideration and increasing evaluation speed.
In most of the conducted research efforts, the sparsity is enforced for network
pruning without any attention to the internal network characteristics such as
unbalanced outputs of the neurons or more specifically the distribution of the
weights and outputs of the neurons. That may cause severe accuracy drop due to
uncontrolled sparsity. In this work, we propose an attention mechanism that
simultaneously controls the sparsity intensity and supervised network pruning
by keeping important information bottlenecks of the network to be active. On
CIFAR-10, the proposed method outperforms the best baseline method by 6% and
reduced the accuracy drop by 2.6x at the same level of sparsity.
