  We consider the on-line predictive version of the standard problem of linear
regression; the goal is to predict each consecutive response given the
corresponding explanatory variables and all the previous observations. The
standard treatment of prediction in linear regression analysis has two
drawbacks: (1) the classical prediction intervals guarantee that the
probability of error is equal to the nominal significance level $\varepsilon$,
but this property per se does not imply that the long-run frequency of error is
close to $\varepsilon$; (2) it is not suitable for prediction of complex
systems as it assumes that the number of observations exceeds the number of
parameters. We state a general result showing that in the on-line protocol the
frequency of error for the classical prediction intervals does equal the
nominal significance level, up to statistical fluctuations. We also describe
alternative regression models in which informative prediction intervals can be
found before the number of observations exceeds the number of parameters. One
of these models, which only assumes that the observations are independent and
identically distributed, is popular in machine learning but greatly underused
in the statistical theory of regression.
