  Ensembles are popular methods for solving practical supervised learning
problems. They reduce the risk of having underperforming models in
production-grade software. Although critical, methods for learning
heterogeneous regression ensembles have not been proposed at large scale,
whereas in classical ML literature, stacking, cascading and voting are mostly
restricted to classification problems. Regression poses distinct learning
challenges that may result in poor performance, even when using well
established homogeneous ensemble schemas such as bagging or boosting.
  In this paper, we introduce MetaBags, a novel, practically useful stacking
framework for regression. MetaBags is a meta-learning algorithm that learns a
set of meta-decision trees designed to select one base model (i.e. expert) for
each query, and focuses on inductive bias reduction. A set of meta-decision
trees are learned using different types of meta-features, specially created for
this purpose - to then be bagged at meta-level. This procedure is designed to
learn a model with a fair bias-variance trade-off, and its improvement over
base model performance is correlated with the prediction diversity of different
experts on specific input space subregions. The proposed method and
meta-features are designed in such a way that they enable good predictive
performance even in subregions of space which are not adequately represented in
the available training data.
  An exhaustive empirical testing of the method was performed, evaluating both
generalization error and scalability of the approach on synthetic, open and
real-world application datasets. The obtained results show that our method
significantly outperforms existing state-of-the-art approaches.
