  Several tensor networks are built of isometric tensors, i.e. tensors
satisfying $W^\dagger W = \mathrm{I}$. Prominent examples include matrix
product states (MPS) in canonical form, the multiscale entanglement
renormalization ansatz (MERA), and quantum circuits in general, such as those
needed in state preparation and quantum variational eigensolvers. We show how
gradient-based optimization methods on Riemannian manifolds can be used to
optimize tensor networks of isometries to represent e.g. ground states of 1D
quantum Hamiltonians. We discuss the geometry of Grassmann and Stiefel
manifolds, the Riemannian manifolds of isometric tensors, and review how
state-of-the-art optimization methods like nonlinear conjugate gradient and
quasi-Newton algorithms can be implemented in this context. We apply these
methods in the context of infinite MPS and MERA, and show benchmark results in
which they outperform the best previously-known optimization methods, which are
tailor-made for those specific variational classes. We also provide open-source
implementations of our algorithms.
