  We consider stochastic optimization problems where the objective depends on
some parameter, as commonly found in hyperparameter optimization for instance.
We investigate the behavior of the derivatives of the iterates of Stochastic
Gradient Descent (SGD) with respect to that parameter and show that they are
driven by an inexact SGD recursion on a different objective function, perturbed
by the convergence of the original SGD. This enables us to establish that the
derivatives of SGD converge to the derivative of the solution mapping in terms
of mean squared error whenever the objective is strongly convex. Specifically,
we demonstrate that with constant step-sizes, these derivatives stabilize
within a noise ball centered at the solution derivative, and that with
vanishing step-sizes they exhibit $O(\log(k)^2 / k)$ convergence rates.
Additionally, we prove exponential convergence in the interpolation regime. Our
theoretical findings are illustrated by numerical experiments on synthetic
tasks.
