  Physics-Informed Neural Networks (PINNs) have gained popularity in scientific
computing in recent years. However, they often fail to achieve the same level
of accuracy as classical methods in solving differential equations. In this
paper, we identify two sources of this issue in the case of Cauchy problems:
the use of $L^2$ residuals as objective functions and the approximation gap of
neural networks. We show that minimizing the sum of $L^2$ residual and initial
condition error is not sufficient to guarantee the true solution, as this loss
function does not capture the underlying dynamics. Additionally, neural
networks are not capable of capturing singularities in the solutions due to the
non-compactness of their image sets. This, in turn, influences the existence of
global minima and the regularity of the network. We demonstrate that when the
global minimum does not exist, machine precision becomes the predominant source
of achievable error in practice. We also present numerical experiments in
support of our theoretical claims.
