  Neural network (NN) designed for challenging machine learning tasks is in
general a highly nonlinear mapping that contains massive variational
parameters. High complexity of NN, if unbounded or unconstrained, might
unpredictably cause severe issues including over-fitting, loss of
generalization power, and unbearable cost of hardware. In this work, we propose
a general compression scheme that significantly reduces the variational
parameters of NN by encoding them to deep automatically-differentiable tensor
network (ADTN) that contains exponentially-fewer free parameters. Superior
compression performance of our scheme is demonstrated on several
widely-recognized NN's (FC-2, LeNet-5, AlextNet, ZFNet and VGG-16) and datasets
(MNIST, CIFAR-10 and CIFAR-100). For instance, we compress two linear layers in
VGG-16 with approximately $10^{7}$ parameters to two ADTN's with just 424
parameters, where the testing accuracy on CIFAR-10 is improved from $90.17 \%$
to $91.74\%$. Our work suggests TN as an exceptionally efficient mathematical
structure for representing the variational parameters of NN's, which exhibits
superior compressibility over the commonly-used matrices and multi-way arrays.
