  It is well understood that neural networks with carefully hand-picked weights
provide powerful function approximation and that they can be successfully
trained in over-parametrized regimes. Since over-parametrization ensures zero
training error, these two theories are not immediately compatible. Recent work
uses the smoothness that is required for approximation results to extend a
neural tangent kernel (NTK) optimization argument to an under-parametrized
regime and show direct approximation bounds for networks trained by gradient
flow. Since gradient flow is only an idealization of a practical method, this
paper establishes analogous results for networks trained by gradient descent.
