  Many mathematical convergence results for gradient descent (GD) based
algorithms employ the assumption that the GD process is (almost surely) bounded
and, also in concrete numerical simulations, divergence of the GD process may
slow down, or even completely rule out, convergence of the error function. In
practical relevant learning problems, it thus seems to be advisable to design
the ANN architectures in a way so that GD optimization processes remain
bounded. The property of the boundedness of GD processes for a given learning
problem seems, however, to be closely related to the existence of minimizers in
the optimization landscape and, in particular, GD trajectories may escape to
infinity if the infimum of the error function (objective function) is not
attained in the optimization landscape. This naturally raises the question of
the existence of minimizers in the optimization landscape and, in the situation
of shallow residual ANNs with multi-dimensional input layers and
multi-dimensional hidden layers with the ReLU activation, the main result of
this work answers this question affirmatively for a general class of loss
functions and all continuous target functions. In our proof of this statement,
we propose a kind of closure of the search space, where the limits are called
generalized responses, and, thereafter, we provide sufficient criteria for the
loss function and the underlying probability distribution which ensure that all
additional artificial generalized responses are suboptimal which finally allows
us to conclude the existence of minimizers in the optimization landscape.
