  Deep neural networks have achieved remarkable success in a wide range of
practical problems. However, due to the inherent large parameter space, deep
models are notoriously prone to overfitting and difficult to be deployed in
portable devices with limited memory. In this paper, we propose an iterative
hard thresholding (IHT) approach to train Skinny Deep Neural Networks (SDNNs).
An SDNN has much fewer parameters yet can achieve competitive or even better
performance than its full CNN counterpart. More concretely, the IHT approach
trains an SDNN through following two alternative phases: (I) perform hard
thresholding to drop connections with small activations and fine-tune the other
significant filters; (II)~re-activate the frozen connections and train the
entire network to improve its overall discriminative capability. We verify the
superiority of SDNNs in terms of efficiency and classification performance on
four benchmark object recognition datasets, including CIFAR-10, CIFAR-100,
MNIST and ImageNet. Experimental results clearly demonstrate that IHT can be
applied for training SDNN based on various CNN architectures such as NIN and
AlexNet.
