  Most neural network pruning methods, such as filter-level and layer-level
prunings, prune the network model along one dimension (depth, width, or
resolution) solely to meet a computational budget. However, such a pruning
policy often leads to excessive reduction of that dimension, thus inducing a
huge accuracy loss. To alleviate this issue, we argue that pruning should be
conducted along three dimensions comprehensively. For this purpose, our pruning
framework formulates pruning as an optimization problem. Specifically, it first
casts the relationships between a certain model's accuracy and
depth/width/resolution into a polynomial regression and then maximizes the
polynomial to acquire the optimal values for the three dimensions. Finally, the
model is pruned along the three optimal dimensions accordingly. In this
framework, since collecting too much data for training the regression is very
time-costly, we propose two approaches to lower the cost: 1) specializing the
polynomial to ensure an accurate regression even with less training data; 2)
employing iterative pruning and fine-tuning to collect the data faster.
Extensive experiments show that our proposed algorithm surpasses
state-of-the-art pruning algorithms and even neural architecture search-based
algorithms.
