  The Physics-Informed Neural Network (PINN) approach is a new and promising
way to solve partial differential equations using deep learning. The $L^2$
Physics-Informed Loss is the de-facto standard in training Physics-Informed
Neural Networks. In this paper, we challenge this common practice by
investigating the relationship between the loss function and the approximation
quality of the learned solution. In particular, we leverage the concept of
stability in the literature of partial differential equation to study the
asymptotic behavior of the learned solution as the loss approaches zero. With
this concept, we study an important class of high-dimensional non-linear PDEs
in optimal control, the Hamilton-Jacobi-Bellman(HJB) Equation, and prove that
for general $L^p$ Physics-Informed Loss, a wide class of HJB equation is stable
only if $p$ is sufficiently large. Therefore, the commonly used $L^2$ loss is
not suitable for training PINN on those equations, while $L^{\infty}$ loss is a
better choice. Based on the theoretical insight, we develop a novel PINN
training algorithm to minimize the $L^{\infty}$ loss for HJB equations which is
in a similar spirit to adversarial training. The effectiveness of the proposed
algorithm is empirically demonstrated through experiments. Our code is released
at https://github.com/LithiumDA/L_inf-PINN.
