  Under mild assumptions stochastic gradient methods asymptotically achieve an
optimal rate of convergence if the arithmetic mean of all iterates is returned
as an approximate optimal solution. However, in the absence of stochastic
noise, the arithmetic mean of all iterates converges considerably slower to the
optimal solution than the iterates themselves. And also in the presence of
noise, when a finite termination of the stochastic gradient method is
considered, the arithmetic mean is not necessarily the best possible
approximation to the unknown optimal solution. This paper aims at identifying
optimal strategies in a particularly simple case, the minimization of a
strongly convex function with i. i. d. noise terms and finite termination.
Explicit formulas for the stochastic error and the optimization error are
derived in dependence of certain parameters of the SGD method. The aim was to
choose parameters such that both stochastic error and optimization error are
reduced compared to arithmetic averaging. This aim could not be achieved;
however, by allowing a slight increase of the stochastic error it was possible
to select the parameters such that a significant reduction of the optimization
error could be achieved. This reduction of the optimization error has a strong
effect on the approximate solution generated by the stochastic gradient method
in case that only a moderate number of iterations is used or when the initial
error is large. The numerical examples confirm the theoretical results and
suggest that a generalization to non-quadratic objective functions may be
possible.
