  We introduce deep tensor networks, which are exponentially wide neural
networks based on the tensor network representation of the weight matrices. We
evaluate the proposed method on the image classification (MNIST, FashionMNIST)
and sequence prediction (cellular automata) tasks. In the image classification
case, deep tensor networks improve our matrix product state baselines and
achieve 0.49% error rate on MNIST and 8.3% error rate on FashionMNIST. In the
sequence prediction case, we demonstrate an exponential improvement in the
number of parameters compared to the one-layer tensor network methods. In both
cases, we discuss the non-uniform and the uniform tensor network models and
show that the latter generalizes well to different input sizes.
