  Gradient descent (GD) methods for the training of artificial neural networks
(ANNs) belong nowadays to the most heavily employed computational schemes in
the digital world. Despite the compelling success of such methods, it remains
an open problem to provide a rigorous theoretical justification for the success
of GD methods in the training of ANNs. The main difficulty is that the
optimization risk landscapes associated to ANNs usually admit many non-optimal
critical points (saddle points as well as non-global local minima) whose risk
values are strictly larger than the optimal risk value. It is a key
contribution of this article to overcome this obstacle in certain simplified
shallow ANN training situations. In such simplified ANN training scenarios we
prove that the gradient flow (GF) dynamics with only one random initialization
overcomes with high probability all bad non-global local minima (all non-global
local minima whose risk values are much larger than the risk value of the
global minima) and converges with high probability to a good critical point (a
critical point whose risk value is very close to the optimal risk value of the
global minima). This analysis allows us to establish convergence in probability
to zero of the risk value of the GF trajectories with convergence rates as the
ANN training time and the width of the ANN increase to infinity. We complement
the analytical findings of this work with extensive numerical simulations for
shallow and deep ANNs: All these numerical simulations strongly suggest that
with high probability the considered GD method (stochastic GD or Adam)
overcomes all bad non-global local minima, does not converge to a global
minimum, but does converge to a good non-optimal critical point whose risk
value is very close to the optimal risk value.
