  To accelerate DNNs inference, low-rank approximation has been widely adopted
because of its solid theoretical rationale and efficient implementations.
Several previous works attempted to directly approximate a pre-trained model by
low-rank decomposition; however, small approximation errors in parameters can
ripple over a large prediction loss. Apparently, it is not optimal to separate
low-rank approximation from training. Unlike previous works, this paper
integrates low rank approximation and regularization into the training process.
We propose Trained Rank Pruning (TRP), which alternates between low rank
approximation and training. TRP maintains the capacity of the original network
while imposing low-rank constraints during training. A nuclear regularization
optimized by stochastic sub-gradient descent is utilized to further promote low
rank in TRP. Networks trained with TRP has a low-rank structure in nature, and
is approximated with negligible performance loss, thus eliminating fine-tuning
after low rank approximation. The proposed method is comprehensively evaluated
on CIFAR-10 and ImageNet, outperforming previous compression counterparts using
low rank approximation. Our code is available at:
https://github.com/yuhuixu1993/Trained-Rank-Pruning.
