  It has been postulated and observed in practice that for prediction problems
in which covariate data can be naturally partitioned into clusters, ensembling
algorithms based on suitably aggregating models trained on individual clusters
often perform substantially better than methods that ignore the clustering
structure in the data. In this paper, we provide theoretical support to these
empirical observations by asymptotically analyzing linear least squares and
random forest regressions under a linear model. Our main results demonstrate
that the benefit of ensembling compared to training a single model on the
entire data, often termed 'merging', might depend on the underlying bias and
variance interplay of the individual predictors to be aggregated. In
particular, under both fixed and high dimensional linear models, we show that
merging is asymptotically superior to optimal ensembling techniques for linear
least squares regression due to the unbiased nature of least squares
prediction. In contrast, for random forest regression under fixed dimensional
linear models, our bounds imply a strict benefit of ensembling over merging.
Finally, we also present numerical experiments to verify the validity of our
asymptotic results across different situations.
