  We propose energy natural gradient descent, a natural gradient method with
respect to a Hessian-induced Riemannian metric as an optimization algorithm for
physics-informed neural networks (PINNs) and the deep Ritz method. As a main
motivation we show that the update direction in function space resulting from
the energy natural gradient corresponds to the Newton direction modulo an
orthogonal projection onto the model's tangent space. We demonstrate
experimentally that energy natural gradient descent yields highly accurate
solutions with errors several orders of magnitude smaller than what is obtained
when training PINNs with standard optimizers like gradient descent or Adam,
even when those are allowed significantly more computation time.
