  We prove linear convergence of gradient descent to a global optimum for the
training of deep residual networks with constant layer width and smooth
activation function. We show that if the trained weights, as a function of the
layer index, admit a scaling limit as the depth increases, then the limit has
finite $p-$variation with $p=2$. Proofs are based on non-asymptotic estimates
for the loss function and for norms of the network weights along the gradient
descent path. We illustrate the relevance of our theoretical results to
practical settings using detailed numerical experiments on supervised learning
problems.
