  Stacking regressions is an ensemble technique that forms linear combinations
of different regression estimators to enhance predictive accuracy. The
conventional approach uses cross-validation data to generate predictions from
the constituent estimators, and least-squares with nonnegativity constraints to
learn the combination weights. In this paper, we learn these weights
analogously by minimizing an estimate of the population risk subject to a
nonnegativity constraint. When the constituent estimators are linear
least-squares projections onto nested subspaces separated by at least three
dimensions, we show that thanks to a shrinkage effect, the resulting stacked
estimator has strictly smaller population risk than best single estimator among
them. Here "best" refers to an estimator that minimizes a model selection
criterion such as AIC or BIC. In other words, in this setting, the best single
estimator is inadmissible. Because the optimization problem can be reformulated
as isotonic regression, the stacked estimator requires the same order of
computation as the best single estimator, making it an attractive alternative
in terms of both performance and implementation.
