  We prove that the gradient descent training of a two-layer neural network on
empirical or population risk may not decrease population risk at an order
faster than $t^{-4/(d-2)}$ under mean field scaling. Thus gradient descent
training for fitting reasonably smooth, but truly high-dimensional data may be
subject to the curse of dimensionality. We present numerical evidence that
gradient descent training with general Lipschitz target functions becomes
slower and slower as the dimension increases, but converges at approximately
the same rate in all dimensions when the target function lies in the natural
function space for two-layer ReLU networks.
