  Structured pruning methods are among the effective strategies for extracting
small resource-efficient convolutional neural networks from their dense
counterparts with minimal loss in accuracy. However, most existing methods
still suffer from one or more limitations, that include 1) the need for
training the dense model from scratch with pruning-related parameters embedded
in the architecture, 2) requiring model-specific hyperparameter settings, 3)
inability to include budget-related constraint in the training process, and 4)
instability under scenarios of extreme pruning. In this paper, we present
ChipNet, a deterministic pruning strategy that employs continuous Heaviside
function and a novel crispness loss to identify a highly sparse network out of
an existing dense network. Our choice of continuous Heaviside function is
inspired by the field of design optimization, where the material distribution
task is posed as a continuous optimization problem, but only discrete values (0
or 1) are practically feasible and expected as final outcomes. Our approach's
flexible design facilitates its use with different choices of budget
constraints while maintaining stability for very low target budgets.
Experimental results show that ChipNet outperforms state-of-the-art structured
pruning methods by remarkable margins of up to 16.1% in terms of accuracy.
Further, we show that the masks obtained with ChipNet are transferable across
datasets. For certain cases, it was observed that masks transferred from a
model trained on feature-rich teacher dataset provide better performance on the
student dataset than those obtained by directly pruning on the student data
itself.
