  With deep learning's success, a limited number of popular deep nets have been
widely adopted for various vision tasks. However, this usually results in
unnecessarily high complexities and possibly many features of low task utility.
In this paper, we address this problem by introducing a task-dependent deep
pruning framework based on Fisher's Linear Discriminant Analysis (LDA). The
approach can be applied to convolutional, fully-connected, and module-based
deep network structures, in all cases leveraging the high decorrelation of
neuron motifs found in the pre-decision space and cross-layer deconv
dependency. Moreover, we examine our approach's potential in network
architecture search for specific tasks and analyze the influence of our pruning
on model robustness to noises and adversarial attacks. Experimental results on
datasets of generic objects (ImageNet, CIFAR100) as well as domain specific
tasks (Adience, and LFWA) illustrate our framework's superior performance over
state-of-the-art pruning approaches and fixed compact nets (e.g. SqueezeNet,
MobileNet). The proposed method successfully maintains comparable accuracies
even after discarding most parameters (98%-99% for VGG16, up to 82% for the
already compact InceptionNet) and with significant FLOP reductions (83% for
VGG16, up to 64% for InceptionNet). Through pruning, we can also derive
smaller, but more accurate and more robust models suitable for the task.
