  We consider the problem of robustly predicting as well as the best linear
combination of $d$ given functions in least squares regression, and variants of
this problem including constraints on the parameters of the linear combination.
For the ridge estimator and the ordinary least squares estimator, and their
variants, we provide new risk bounds of order $d/n$ without logarithmic factor
unlike some standard results, where $n$ is the size of the training data. We
also provide a new estimator with better deviations in the presence of
heavy-tailed noise. It is based on truncating differences of losses in a
min--max framework and satisfies a $d/n$ risk bound both in expectation and in
deviations. The key common surprising factor of these results is the absence of
exponential moment condition on the output distribution while achieving
exponential deviations. All risk bounds are obtained through a PAC-Bayesian
analysis on truncated differences of losses. Experimental results strongly back
up our truncated min--max estimator.
