  We consider the scenario of supervised learning in Deep Learning (DL)
networks, and exploit the arbitrariness of choice in the Riemannian metric
relative to which the gradient descent flow can be defined (a general fact of
differential geometry). In the standard approach to DL, the gradient flow on
the space of parameters (weights and biases) is defined with respect to the
Euclidean metric. Here instead, we choose the gradient flow with respect to the
Euclidean metric in the output layer of the DL network. This naturally induces
two modified versions of the gradient descent flow in the parameter space, one
adapted for the overparametrized setting, and the other for the
underparametrized setting. In the overparametrized case, we prove that,
provided that a rank condition holds, all orbits of the modified gradient
descent drive the ${\mathcal L}^2$ cost to its global minimum at a uniform
exponential convergence rate; one thereby obtains an a priori stopping time for
any prescribed proximity to the global minimum. We point out relations of the
latter to sub-Riemannian geometry. Moreover, we generalize the above framework
to the situation in which the rank condition does not hold; in particular, we
show that local equilibria can only exist if a rank loss occurs, and that
generically, they are not isolated points, but elements of a critical
submanifold of parameter space.
