  Tensor Networks (TN) offer a powerful framework to efficiently represent very
high-dimensional objects. TN have recently shown their potential for machine
learning applications and offer a unifying view of common tensor decomposition
models such as Tucker, tensor train (TT) and tensor ring (TR). However,
identifying the best tensor network structure from data for a given task is
challenging. In this work, we leverage the TN formalism to develop a generic
and efficient adaptive algorithm to jointly learn the structure and the
parameters of a TN from data. Our method is based on a simple greedy approach
starting from a rank one tensor and successively identifying the most promising
tensor network edges for small rank increments. Our algorithm can adaptively
identify TN structures with small number of parameters that effectively
optimize any differentiable objective function. Experiments on tensor
decomposition, tensor completion and model compression tasks demonstrate the
effectiveness of the proposed algorithm. In particular, our method outperforms
the state-of-the-art evolutionary topology search [Li and Sun, 2020] for tensor
decomposition of images (while being orders of magnitude faster) and finds
efficient tensor network structures to compress neural networks outperforming
popular TT based approaches [Novikov et al., 2015].
