  In this article we investigate blow up phenomena for gradient descent
optimization methods in the training of artificial neural networks (ANNs). Our
theoretical analysis is focused on shallow ANNs with one neuron on the input
layer, one neuron on the output layer, and one hidden layer. For ANNs with ReLU
activation and at least two neurons on the hidden layer we establish the
existence of a target function such that there exists a lower bound for the
risk values of the critical points of the associated risk function which is
strictly greater than the infimum of the image of the risk function. This
allows us to demonstrate that every gradient flow trajectory with an initial
risk smaller than this lower bound diverges. Furthermore, we analyze and
compare various popular types of activation functions with regard to the
divergence of gradient flow trajectories and gradient descent trajectories in
the training of ANNs and with regard to the closely related question concerning
the existence of global minimum points of the risk function.
