  The pressing need to reduce the capacity of deep neural networks has
stimulated the development of network dilution methods and their analysis.
While the ability of $L_1$ and $L_0$ regularization to encourage sparsity is
often mentioned, $L_2$ regularization is seldom discussed in this context. We
present a novel framework for weight pruning by sampling from a probability
function that favors the zeroing of smaller weights. In addition, we examine
the contribution of $L_1$ and $L_2$ regularization to the dynamics of node
pruning while optimizing for weight pruning. We then demonstrate the
effectiveness of the proposed stochastic framework when used together with a
weight decay regularizer on popular classification models in removing 50% of
the nodes in an MLP for MNIST classification, 60% of the filters in VGG-16 for
CIFAR10 classification, and on medical image models in removing 60% of the
channels in a U-Net for instance segmentation and 50% of the channels in CNN
model for COVID-19 detection. For these node-pruned networks, we also present
competitive weight pruning results that are only slightly less accurate than
the original, dense networks.
