  We study the problem of solving strongly convex and smooth unconstrained
optimization problems using stochastic first-order algorithms. We devise a
novel algorithm, referred to as \emph{Recursive One-Over-T SGD}
(\textsf{ROOT-SGD}), based on an easily implementable, recursive averaging of
past stochastic gradients. We prove that it simultaneously achieves
state-of-the-art performance in both a finite-sample, nonasymptotic sense and
an asymptotic sense. On the nonasymptotic side, we prove risk bounds on the
last iterate of \textsf{ROOT-SGD} with leading-order terms that match the
optimal statistical risk with a unity pre-factor, along with a higher-order
term that scales at the sharp rate of $O(n^{-3/2})$ under the Lipschitz
condition on the Hessian matrix. On the asymptotic side, we show that when a
mild, one-point Hessian continuity condition is imposed, the rescaled last
iterate of (multi-epoch) \textsf{ROOT-SGD} converges asymptotically to a
Gaussian limit with the Cram\'{e}r-Rao optimal asymptotic covariance, for a
broad range of step-size choices.
