  High-order optimization methods, including Newton's method and its variants
as well as alternating minimization methods, dominate the optimization
algorithms for tensor decompositions and tensor networks. These tensor methods
are used for data analysis and simulation of quantum systems. In this work, we
introduce AutoHOOT, the first automatic differentiation (AD) framework
targeting at high-order optimization for tensor computations. AutoHOOT takes
input tensor computation expressions and generates optimized derivative
expressions. In particular, AutoHOOT contains a new explicit Jacobian / Hessian
expression generation kernel whose outputs maintain the input tensors'
granularity and are easy to optimize. The expressions are then optimized by
both the traditional compiler optimization techniques and specific tensor
algebra transformations. Experimental results show that AutoHOOT achieves
competitive CPU and GPU performance for both tensor decomposition and tensor
network applications compared to existing AD software and other tensor
computation libraries with manually written kernels. The tensor methods
generated by AutoHOOT are also well-parallelizable, and we demonstrate good
scalability on a distributed memory supercomputer.
