  Gradient based optimization algorithms deployed in Machine Learning (ML)
applications are often analyzed and compared by their convergence rates or
regret bounds. While these rates and bounds convey valuable information they
don't always directly translate to stability guarantees. Stability and similar
concepts, like robustness, will become ever more important as we move towards
deploying models in real-time and safety critical systems. In this work we
build upon the results in Gaudio et al. 2021 and Moreu and Annaswamy 2022 for
second order gradient descent when applied to explicitly time varying cost
functions and provide more general stability guarantees. These more general
results can aid in the design and certification of these optimization schemes
so as to help ensure safe and reliable deployment for real-time learning
applications. We also hope that the techniques provided here will stimulate and
cross-fertilize the analysis that occurs on the same algorithms from the online
learning and stochastic optimization communities.
