  Advanced tensor decomposition, such as Tensor train (TT) and Tensor ring
(TR), has been widely studied for deep neural network (DNN) model compression,
especially for recurrent neural networks (RNNs). However, compressing
convolutional neural networks (CNNs) using TT/TR always suffers significant
accuracy loss. In this paper, we propose a systematic framework for tensor
decomposition-based model compression using Alternating Direction Method of
Multipliers (ADMM). By formulating TT decomposition-based model compression to
an optimization problem with constraints on tensor ranks, we leverage ADMM
technique to systemically solve this optimization problem in an iterative way.
During this procedure, the entire DNN model is trained in the original
structure instead of TT format, but gradually enjoys the desired low tensor
rank characteristics. We then decompose this uncompressed model to TT format
and fine-tune it to finally obtain a high-accuracy TT-format DNN model. Our
framework is very general, and it works for both CNNs and RNNs, and can be
easily modified to fit other tensor decomposition approaches. We evaluate our
proposed framework on different DNN models for image classification and video
recognition tasks. Experimental results show that our ADMM-based TT-format
models demonstrate very high compression performance with high accuracy.
Notably, on CIFAR-100, with 2.3X and 2.4X compression ratios, our models have
1.96% and 2.21% higher top-1 accuracy than the original ResNet-20 and
ResNet-32, respectively. For compressing ResNet-18 on ImageNet, our model
achieves 2.47X FLOPs reduction without accuracy loss.
