  This paper explores the difficulties in solving partial differential
equations (PDEs) using physics-informed neural networks (PINNs). PINNs use
physics as a regularization term in the objective function. However, a drawback
of this approach is the requirement for manual hyperparameter tuning, making it
impractical in the absence of validation data or prior knowledge of the
solution. Our investigations of the loss landscapes and backpropagated
gradients in the presence of physics reveal that existing methods produce
non-convex loss landscapes that are hard to navigate. Our findings demonstrate
that high-order PDEs contaminate backpropagated gradients and hinder
convergence. To address these challenges, we introduce a novel method that
bypasses the calculation of high-order derivative operators and mitigates the
contamination of backpropagated gradients. Consequently, we reduce the
dimension of the search space and make learning PDEs with non-smooth solutions
feasible. Our method also provides a mechanism to focus on complex regions of
the domain. Besides, we present a dual unconstrained formulation based on
Lagrange multiplier method to enforce equality constraints on the model's
prediction, with adaptive and independent learning rates inspired by adaptive
subgradient methods. We apply our approach to solve various linear and
non-linear PDEs.
