  Despite many modern applications of Deep Neural Networks (DNNs), the large
number of parameters in the hidden layers makes them unattractive for
deployment on devices with storage capacity constraints. In this paper we
propose a Data-Driven Low-rank (DDLR) method to reduce the number of parameters
of pretrained DNNs and expedite inference by imposing low-rank structure on the
fully connected layers, while controlling for the overall accuracy and without
requiring any retraining. We pose the problem as finding the lowest rank
approximation of each fully connected layer with given performance guarantees
and relax it to a tractable convex optimization problem. We show that it is
possible to significantly reduce the number of parameters in common DNN
architectures with only a small reduction in classification accuracy. We
compare DDLR with Net-Trim, which is another data-driven DNN compression
technique based on sparsity and show that DDLR consistently produces more
compressed neural networks while maintaining higher accuracy.
