  The Lipschitz constant is an important quantity that arises in analysing the
convergence of gradient-based optimization methods. It is generally unclear how
to estimate the Lipschitz constant of a complex model. Thus, this paper studies
an important problem that may be useful to the broader area of non-convex
optimization. The main result provides a local upper bound on the Lipschitz
constants of a multi-layer feed-forward neural network and its gradient.
Moreover, lower bounds are established as well, which are used to show that it
is impossible to derive global upper bounds for the Lipschitz constants. In
contrast to previous works, we compute the Lipschitz constants with respect to
the network parameters and not with respect to the inputs. These constants are
needed for the theoretical description of many step size schedulers of gradient
based optimization schemes and their convergence analysis. The idea is both
simple and effective. The results are extended to a generalization of neural
networks, continuously deep neural networks, which are described by controlled
ODEs.
