  To enable DNNs on edge devices like mobile phones, low-rank approximation has
been widely adopted because of its solid theoretical rationale and efficient
implementations. Several previous works attempted to directly approximate a
pretrained model by low-rank decomposition; however, small approximation errors
in parameters can ripple over a large prediction loss. As a result, performance
usually drops significantly and a sophisticated effort on fine-tuning is
required to recover accuracy. Apparently, it is not optimal to separate
low-rank approximation from training. Unlike previous works, this paper
integrates low rank approximation and regularization into the training process.
We propose Trained Rank Pruning (TRP), which alternates between low rank
approximation and training. TRP maintains the capacity of the original network
while imposing low-rank constraints during training. A nuclear regularization
optimized by stochastic sub-gradient descent is utilized to further promote low
rank in TRP. The TRP trained network inherently has a low-rank structure, and
is approximated with negligible performance loss, thus eliminating the
fine-tuning process after low rank decomposition. The proposed method is
comprehensively evaluated on CIFAR-10 and ImageNet, outperforming previous
compression methods using low rank approximation.
