  While training error of most deep neural networks degrades as the depth of
the network increases, residual networks appear to be an exception. We show
that the main reason for this is the Lyapunov stability of the gradient descent
algorithm: for an arbitrarily chosen step size, the equilibria of the gradient
descent are most likely to remain stable for the parametrization of residual
networks. We then present an architecture with a pair of residual networks to
approximate a large class of functions by decomposing them into a convex and a
concave part. Some parameters of this model are shown to change little during
training, and this imperfect optimization prevents overfitting the data and
leads to solutions with small Lipschitz constants, while providing clues about
the generalization of other deep networks.
