  In this paper, we study the stochastic optimization problem from a
continuous-time perspective, with a focus on the Stochastic Gradient Descent
with Momentum (SGDM) method. We show that the trajectory of SGDM, despite its
stochastic nature, converges to a deterministic second-order Ordinary
Differential Equation (ODE) in $L_2$-norm, as the stepsize goes to zero. The
connection between the ODE and the algorithm results in delightful patterns in
the discrete-time convergence analysis. More specifically, we develop
convergence results for the ODE through a Lyapunov function, and translate the
whole argument to the discrete-time case. This approach yields a novel anytime
convergence guarantee for stochastic gradient methods. Precisely, we prove that
the sequence $\{ x_k \}$ governed by running SGDM on a smooth convex function
$f$ satisfies \begin{align*}
  \mathbb{P}\left(\text{for any $k$},\;f (x_k) - f^* \le
C\left(1+\log\frac{1}{\beta}\right)\frac{\log k}{\sqrt{k}}\right)\ge 1-\beta
\end{align*} for any $\beta$, where $f^*=\min_{x\in\mathbb{R}^n} f(x)$, and $C$
is a constant. Our contribution is significant in that it better captures the
convergence behavior across the entire trajectory of the algorithm, rather than
at a single iterate.
