  A physics-informed neural network (PINN) uses physics-augmented loss
functions, e.g., incorporating the residual term from governing partial
differential equations (PDEs), to ensure its output is consistent with
fundamental physics laws. However, it turns out to be difficult to train an
accurate PINN model for many problems in practice. In this paper, we present a
novel perspective of the merits of learning in sinusoidal spaces with PINNs. By
analyzing behavior at model initialization, we first show that a PINN of
increasing expressiveness induces an initial bias around flat output functions.
Notably, this initial solution can be very close to satisfying many physics
PDEs, i.e., falling into a local minimum of the PINN loss that only minimizes
PDE residuals, while still being far from the true solution that jointly
minimizes PDE residuals and the initial and/or boundary conditions. It is
difficult for gradient descent optimization to escape from such a local minimum
trap, often causing the training to stall. We then prove that the sinusoidal
mapping of inputs, in an architecture we label as sf-PINN, is effective to
increase input gradient variability, thus avoiding being trapped in such
deceptive local minimum. The level of variability can be effectively modulated
to match high-frequency patterns in the problem at hand. A key facet of this
paper is the comprehensive empirical study that demonstrates the efficacy of
learning in sinusoidal spaces with PINNs for a wide range of forward and
inverse modelling problems spanning multiple physics domains.
