  Features in machine learning problems are often time-varying and may be
related to outputs in an algebraic or dynamical manner. The dynamic nature of
these machine learning problems renders current higher order accelerated
gradient descent methods unstable or weakens their convergence guarantees.
Inspired by methods employed in adaptive control, this paper proposes new
algorithms for the case when time-varying features are present, and
demonstrates provable performance guarantees. In particular, we develop a
unified variational perspective within a continuous time algorithm. This
variational perspective includes higher order learning concepts and
normalization, both of which stem from adaptive control, and allows stability
to be established for dynamical machine learning problems where time-varying
features are present. These higher order algorithms are also examined for
provably correct learning in adaptive control and identification. Simulations
are provided to verify the theoretical results.
