  Deep learning algorithms - typically consisting of a class of deep neural
networks trained by a stochastic gradient descent (SGD) optimization method -
are nowadays the key ingredients in many artificial intelligence (AI) systems
and have revolutionized our ways of working and living in modern societies. For
example, SGD methods are used to train powerful large language models (LLMs)
such as versions of ChatGPT and Gemini, SGD methods are employed to create
successful generative AI based text-to-image creation models such as
Midjourney, DALL-E, and Stable Diffusion, but SGD methods are also used to
train DNNs to approximately solve scientific models such as partial
differential equation (PDE) models from physics and biology and optimal control
and stopping problems from engineering. It is known that the plain vanilla
standard SGD method fails to converge even in the situation of several convex
optimization problems if the learning rates are bounded away from zero.
However, in many practical relevant training scenarios, often not the plain
vanilla standard SGD method but instead adaptive SGD methods such as the
RMSprop and the Adam optimizers, in which the learning rates are modified
adaptively during the training process, are employed. This naturally rises the
question whether such adaptive optimizers, in which the learning rates are
modified adaptively during the training process, do converge in the situation
of non-vanishing learning rates. In this work we answer this question
negatively by proving that adaptive SGD methods such as the popular Adam
optimizer fail to converge to any possible random limit point if the learning
rates are asymptotically bounded away from zero. In our proof of this
non-convergence result we establish suitable pathwise a priori bounds for a
class of accelerated and adaptive SGD methods, which are also of independent
interest.
