  Stochastic optimization has found wide applications in minimizing objective
functions in machine learning, which motivates a lot of theoretical studies to
understand its practical success. Most of existing studies focus on the
convergence of optimization errors, while the generalization analysis of
stochastic optimization is much lagging behind. This is especially the case for
nonconvex and nonsmooth problems often encountered in practice. In this paper,
we initialize a systematic stability and generalization analysis of stochastic
optimization on nonconvex and nonsmooth problems. We introduce novel
algorithmic stability measures and establish their quantitative connection on
the gap between population gradients and empirical gradients, which is then
further extended to study the gap between the Moreau envelope of the empirical
risk and that of the population risk. To our knowledge, these quantitative
connection between stability and generalization in terms of either gradients or
Moreau envelopes have not been studied in the literature. We introduce a class
of sampling-determined algorithms, for which we develop bounds for three
stability measures. Finally, we apply these discussions to derive error bounds
for stochastic gradient descent and its adaptive variant, where we show how to
achieve an implicit regularization by tuning the step sizes and the number of
iterations.
