  In this paper, we study the importance of pruning in Deep Networks (DNs) and
the yin & yang relationship between (1) pruning highly overparametrized DNs
that have been trained from random initialization and (2) training small DNs
that have been "cleverly" initialized. As in most cases practitioners can only
resort to random initialization, there is a strong need to develop a grounded
understanding of DN pruning. Current literature remains largely empirical,
lacking a theoretical understanding of how pruning affects DNs' decision
boundary, how to interpret pruning, and how to design corresponding principled
pruning techniques. To tackle those questions, we propose to employ recent
advances in the theoretical analysis of Continuous Piecewise Affine (CPA) DNs.
From this perspective, we will be able to detect the early-bird (EB) ticket
phenomenon, provide interpretability into current pruning techniques, and
develop a principled pruning strategy. In each step of our study, we conduct
extensive experiments supporting our claims and results; while our main goal is
to enhance the current understanding towards DN pruning instead of developing a
new pruning method, our spline pruning criteria in terms of layerwise and
global pruning is on par with or even outperforms state-of-the-art pruning
methods.
