  Ensembling is among the most popular tools in machine learning (ML) due to
its effectiveness in minimizing variance and thus improving generalization.
Most ensembling methods for black-box base learners fall under the umbrella of
"stacked generalization," namely training an ML algorithm that takes the
inferences from the base learners as input. While stacking has been widely
applied in practice, its theoretical properties are poorly understood. In this
paper, we prove a novel result, showing that choosing the best stacked
generalization from a (finite or finite-dimensional) family of stacked
generalizations based on cross-validated performance does not perform "much
worse" than the oracle best. Our result strengthens and significantly extends
the results in Van der Laan et al. (2007). Inspired by the theoretical
analysis, we further propose a particular family of stacked generalizations in
the context of probabilistic forecasting, each one with a different sensitivity
for how much the ensemble weights are allowed to vary across items, timestamps
in the forecast horizon, and quantiles. Experimental results demonstrate the
performance gain of the proposed method.
