  In this paper, we study the statistical limits in terms of Sobolev norms of
gradient descent for solving inverse problem from randomly sampled noisy
observations using a general class of objective functions. Our class of
objective functions includes Sobolev training for kernel regression, Deep Ritz
Methods (DRM), and Physics Informed Neural Networks (PINN) for solving elliptic
partial differential equations (PDEs) as special cases. We consider a
potentially infinite-dimensional parameterization of our model using a suitable
Reproducing Kernel Hilbert Space and a continuous parameterization of problem
hardness through the definition of kernel integral operators. We prove that
gradient descent over this objective function can also achieve statistical
optimality and the optimal number of passes over the data increases with sample
size. Based on our theory, we explain an implicit acceleration of using a
Sobolev norm as the objective function for training, inferring that the optimal
number of epochs of DRM becomes larger than the number of PINN when both the
data size and the hardness of tasks increase, although both DRM and PINN can
achieve statistical optimality.
