  Neural networks can be used to learn the solution of partial differential
equations (PDEs) on arbitrary domains without requiring a computational mesh.
Common approaches integrate differential operators in training neural networks
using a structured loss function. The most common training algorithm for neural
networks is backpropagation which relies on the gradient of the loss function
with respect to the parameters of the network. In this work, we characterize
the difficulty of training neural networks on physics by investigating the
impact of differential operators in corrupting the back propagated gradients.
Particularly, we show that perturbations present in the output of a neural
network model during early stages of training lead to higher levels of noise in
a structured loss function that is composed of high-order differential
operators. These perturbations consequently corrupt the back-propagated
gradients and impede convergence. We mitigate this issue by introducing
auxiliary flux parameters to obtain a system of first-order differential
equations. We formulate a non-linear unconstrained optimization problem using
the augmented Lagrangian method that properly constrains the boundary
conditions and adaptively focus on regions of higher gradients that are
difficult to learn. We apply our approach to learn the solution of various
benchmark PDE problems and demonstrate orders of magnitude improvement over
existing approaches.
