  Physics-informed neural networks (PINNs) offer a promising avenue for
tackling both forward and inverse problems in partial differential equations
(PDEs) by incorporating deep learning with fundamental physics principles.
Despite their remarkable empirical success, PINNs have garnered a reputation
for their notorious training challenges across a spectrum of PDEs. In this
work, we delve into the intricacies of PINN optimization from a neural
architecture perspective. Leveraging the Neural Tangent Kernel (NTK), our study
reveals that Gaussian activations surpass several alternate activations when it
comes to effectively training PINNs. Building on insights from numerical linear
algebra, we introduce a preconditioned neural architecture, showcasing how such
tailored architectures enhance the optimization process. Our theoretical
findings are substantiated through rigorous validation against established PDEs
within the scientific literature.
