  Model compression is a crucial part of deploying neural networks (NNs),
especially when the memory and storage of computing devices are limited in many
applications. This paper focuses on two model compression techniques: low-rank
approximation and weight pruning in neural networks, which are very popular
nowadays. However, training NN with low-rank approximation and weight pruning
always suffers significant accuracy loss and convergence issues. In this paper,
a holistic framework is proposed for model compression from a novel perspective
of nonconvex optimization by designing an appropriate objective function. Then,
we introduce NN-BCD, a block coordinate descent (BCD) algorithm to solve the
nonconvex optimization. One advantage of our algorithm is that an efficient
iteration scheme can be derived with closed-form, which is gradient-free.
Therefore, our algorithm will not suffer from vanishing/exploding gradient
problems. Furthermore, with the Kurdyka-{\L}ojasiewicz (K{\L}) property of our
objective function, we show that our algorithm globally converges to a critical
point at the rate of O(1/k), where k denotes the number of iterations. Lastly,
extensive experiments with tensor train decomposition and weight pruning
demonstrate the efficiency and superior performance of the proposed framework.
Our code implementation is available at https://github.com/ChenyangLi-97/NN-BCD
