  Stochastic gradient methods enable learning probabilistic models from large
amounts of data. While large step-sizes (learning rates) have shown to be best
for least-squares (e.g., Gaussian noise) once combined with parameter
averaging, these are not leading to convergent algorithms in general. In this
paper, we consider generalized linear models, that is, conditional models based
on exponential families. We propose averaging moment parameters instead of
natural parameters for constant-step-size stochastic gradient descent. For
finite-dimensional models, we show that this can sometimes (and surprisingly)
lead to better predictions than the best linear model. For infinite-dimensional
models, we show that it always converges to optimal predictions, while
averaging natural parameters never does. We illustrate our findings with
simulations on synthetic data and classical benchmarks with many observations.
