  Whilst the Universal Approximation Theorem guarantees the existence of
approximations to Sobolev functions -- the natural function spaces for PDEs --
by Neural Networks (NNs) of sufficient size, low-regularity solutions may lead
to poor approximations in practice. For example, classical fully-connected
feed-forward NNs fail to approximate continuous functions whose gradient is
discontinuous when employing strong formulations like in Physics Informed
Neural Networks (PINNs). In this article, we propose the use of
regularity-conforming neural networks, where a priori information on the
regularity of solutions to PDEs can be employed to construct proper
architectures. We illustrate the potential of such architectures via a
two-dimensional (2D) transmission problem, where the solution may admit
discontinuities in the gradient across interfaces, as well as power-like
singularities at certain points. In particular, we formulate the weak
transmission problem in a PINNs-like strong formulation with interface and
continuity conditions. Such architectures are partially explainable;
discontinuities are explicitly described, allowing the introduction of novel
terms into the loss function. We demonstrate via several model problems in one
and two dimensions the advantages of using regularity-conforming architectures
in contrast to classical architectures. The ideas presented in this article
easily extend to problems in higher dimensions.
