  This paper explores challenges in training Physics-Informed Neural Networks
(PINNs), emphasizing the role of the loss landscape in the training process. We
examine difficulties in minimizing the PINN loss function, particularly due to
ill-conditioning caused by differential operators in the residual term. We
compare gradient-based optimizers Adam, L-BFGS, and their combination
Adam+L-BFGS, showing the superiority of Adam+L-BFGS, and introduce a novel
second-order optimizer, NysNewton-CG (NNCG), which significantly improves PINN
performance. Theoretically, our work elucidates the connection between
ill-conditioned differential operators and ill-conditioning in the PINN loss
and shows the benefits of combining first- and second-order optimization
methods. Our work presents valuable insights and more powerful optimization
strategies for training PINNs, which could improve the utility of PINNs for
solving difficult partial differential equations.
