  Low-rankness plays an important role in traditional machine learning, but is
not so popular in deep learning. Most previous low-rank network compression
methods compress the networks by approximating pre-trained models and
re-training. However, the optimal solution in the Euclidean space may be quite
different from the one in the low-rank manifold. A well-pre-trained model is
not a good initialization for the model with low-rank constraints. Thus, the
performance of a low-rank compressed network degrades significantly. Compared
to other network compression methods such as pruning, low-rank methods attracts
less attention in recent years. In this paper, we devise a new training method,
low-rank projection with energy transfer (LRPET), that trains low-rank
compressed networks from scratch and achieves competitive performance. First,
we propose to alternately perform stochastic gradient descent training and
projection onto the low-rank manifold. Compared to re-training on the compact
model, this enables full utilization of model capacity since solution space is
relaxed back to Euclidean space after projection. Second, the matrix energy
(the sum of squares of singular values) reduction caused by projection is
compensated by energy transfer. We uniformly transfer the energy of the pruned
singular values to the remaining ones. We theoretically show that energy
transfer eases the trend of gradient vanishing caused by projection. Third, we
propose batch normalization (BN) rectification to cut off its effect on the
optimal low-rank approximation of the weight matrix, which further improves the
performance. Comprehensive experiments on CIFAR-10 and ImageNet have justified
that our method is superior to other low-rank compression methods and also
outperforms recent state-of-the-art pruning methods. Our code is available at
https://github.com/BZQLin/LRPET.
