  In this paper, we first reinvestigate the convergence of vanilla SGD method
in the sense of $L^2$ under more general learning rates conditions and a more
general convex assumption, which relieves the conditions on learning rates and
do not need the problem to be strongly convex. Then, by taking advantage of the
Lyapunov function technique, we present the convergence of the momentum SGD and
Nesterov accelerated SGD methods for the convex and non-convex problem under
$L$-smooth assumption that extends the bounded gradient limitation to a certain
extent. The convergence of time averaged SGD was also analyzed.
