  In the past several years, the last-iterate convergence of the Stochastic
Gradient Descent (SGD) algorithm has triggered people's interest due to its
good performance in practice but lack of theoretical understanding. For
Lipschitz convex functions, different works have established the optimal
$O(\log(1/\delta)\log T/\sqrt{T})$ or $O(\sqrt{\log(1/\delta)/T})$
high-probability convergence rates for the final iterate, where $T$ is the time
horizon and $\delta$ is the failure probability. However, to prove these
bounds, all the existing works are either limited to compact domains or require
almost surely bounded noises. It is natural to ask whether the last iterate of
SGD can still guarantee the optimal convergence rate but without these two
restrictive assumptions. Besides this important question, there are still lots
of theoretical problems lacking an answer. For example, compared with the
last-iterate convergence of SGD for non-smooth problems, only few results for
smooth optimization have yet been developed. Additionally, the existing results
are all limited to a non-composite objective and the standard Euclidean norm.
It still remains unclear whether the last-iterate convergence can be provably
extended to wider composite optimization and non-Euclidean norms. In this work,
to address the issues mentioned above, we revisit the last-iterate convergence
of stochastic gradient methods and provide the first unified way to prove the
convergence rates both in expectation and in high probability to accommodate
general domains, composite objectives, non-Euclidean norms, Lipschitz
conditions, smoothness, and (strong) convexity simultaneously. Additionally, we
extend our analysis to obtain the last-iterate convergence under heavy-tailed
noises.
