  The gradient boosting machine is one of the powerful tools for solving
regression problems. In order to cope with its shortcomings, an approach for
constructing ensembles of gradient boosting models is proposed. The main idea
behind the approach is to use the stacking algorithm in order to learn a
second-level meta-model which can be regarded as a model for implementing
various ensembles of gradient boosting models. First, the linear regression of
the gradient boosting models is considered as a simplest realization of the
meta-model under condition that the linear model is differentiable with respect
to its coefficients (weights). Then it is shown that the proposed approach can
be simply extended on arbitrary differentiable combination models, for example,
on neural networks which are differentiable and can implement arbitrary
functions of gradient boosting models. Various numerical examples illustrate
the proposed approach.
