  Regularization for optimization is a crucial technique to avoid overfitting
in machine learning. In order to obtain the best performance, we usually train
a model by tuning the regularization parameters. It becomes costly, however,
when a single round of training takes significant amount of time. Very
recently, Neu and Rosasco show that if we run stochastic gradient descent (SGD)
on linear regression problems, then by averaging the SGD iterates properly, we
obtain a regularized solution. It left open whether the same phenomenon can be
achieved for other optimization problems and algorithms. In this paper, we
establish an averaging scheme that provably converts the iterates of SGD on an
arbitrary strongly convex and smooth objective function to its regularized
counterpart with an adjustable regularization parameter. Our approaches can be
used for accelerated and preconditioned optimization methods as well. We
further show that the same methods work empirically on more general
optimization objectives including neural networks. In sum, we obtain adjustable
regularization for free for a large class of optimization problems and resolve
an open question raised by Neu and Rosasco.
