  Gradient descent (GD) type optimization schemes are the standard methods to
train artificial neural networks (ANNs) with rectified linear unit (ReLU)
activation. Such schemes can be considered as discretizations of gradient flows
(GFs) associated to the training of ANNs with ReLU activation and most of the
key difficulties in the mathematical convergence analysis of GD type
optimization schemes in the training of ANNs with ReLU activation seem to be
already present in the dynamics of the corresponding GF differential equations.
It is the key subject of this work to analyze such GF differential equations in
the training of ANNs with ReLU activation and three layers (one input layer,
one hidden layer, and one output layer). In particular, in this article we
prove in the case where the target function is possibly multi-dimensional and
continuous and in the case where the probability distribution of the input data
is absolutely continuous with respect to the Lebesgue measure that the risk of
every bounded GF trajectory converges to the risk of a critical point. In
addition, in this article we show in the case of a 1-dimensional affine linear
target function and in the case where the probability distribution of the input
data coincides with the standard uniform distribution that the risk of every
bounded GF trajectory converges to zero if the initial risk is sufficiently
small. Finally, in the special situation where there is only one neuron on the
hidden layer (1-dimensional hidden layer) we strengthen the above named result
for affine linear target functions by proving that that the risk of every (not
necessarily bounded) GF trajectory converges to zero if the initial risk is
sufficiently small.
