  Overparameterized neural networks enjoy great representation power on complex
data, and more importantly yield sufficiently smooth output, which is crucial
to their generalization and robustness. Most existing function approximation
theories suggest that with sufficiently many parameters, neural networks can
well approximate certain classes of functions in terms of the function value.
The neural network themselves, however, can be highly nonsmooth. To bridge this
gap, we take convolutional residual networks (ConvResNets) as an example, and
prove that large ConvResNets can not only approximate a target function in
terms of function value, but also exhibit sufficient first-order smoothness.
Moreover, we extend our theory to approximating functions supported on a
low-dimensional manifold. Our theory partially justifies the benefits of using
deep and wide networks in practice. Numerical experiments on adversarial robust
image classification are provided to support our theory.
