  Pruning neural networks reduces inference time and memory costs. On standard
hardware, these benefits will be especially prominent if coarse-grained
structures, like feature maps, are pruned. We devise two novel saliency-based
methods for second-order structured pruning (SOSP) which include correlations
among all structures and layers. Our main method SOSP-H employs an innovative
second-order approximation, which enables saliency evaluations by fast
Hessian-vector products. SOSP-H thereby scales like a first-order method
despite taking into account the full Hessian. We validate SOSP-H by comparing
it to our second method SOSP-I that uses a well-established Hessian
approximation, and to numerous state-of-the-art methods. While SOSP-H performs
on par or better in terms of accuracy, it has clear advantages in terms of
scalability and efficiency. This allowed us to scale SOSP-H to large-scale
vision tasks, even though it captures correlations across all layers of the
network. To underscore the global nature of our pruning methods, we evaluate
their performance not only by removing structures from a pretrained network,
but also by detecting architectural bottlenecks. We show that our algorithms
allow to systematically reveal architectural bottlenecks, which we then remove
to further increase the accuracy of the networks.
