  Deep learning-based numerical schemes such as Physically Informed Neural
Networks (PINNs) have recently emerged as an alternative to classical numerical
schemes for solving Partial Differential Equations (PDEs). They are very
appealing at first sight because implementing vanilla versions of PINNs based
on strong residual forms is easy, and neural networks offer very high
approximation capabilities. However, when the PDE solutions are low regular, an
expert insight is required to build deep learning formulations that do not
incur in variational crimes. Optimization solvers are also significantly
challenged, and can potentially spoil the final quality of the approximated
solution due to the convergence to bad local minima, and bad generalization
capabilities. In this paper, we present an exhaustive numerical study of the
merits and limitations of these schemes when solutions exhibit low-regularity,
and compare performance with respect to more benign cases when solutions are
very smooth. As a support for our study, we consider singularly perturbed
convection-diffusion problems where the regularity of solutions typically
degrades as certain multiscale parameters go to zero.
