While learning about Gradient Boosting, I haven't heard about any constraints regarding the properties of a "weak classifier" that the method uses to build and ensemble model. However, I could not imagine an application of a GB that uses linear regression, and in fact when I've performed some tests - it doesn't work. I was testing the most standard approach with a gradient of sum of squared residuals and adding the subsequent models together.

The obvious problem is that the residuals from the first model are populated in such manner that there is really no regression line to fit anymore. My another observation is that a sum of subsequent linear regression models can be represented as a single regression model as well (adding all intercepts and corresponding coefficients) so I cannot imagine how that could ever improve the model. The last observation is that a linear regression (the most typical approach) is using sum of squared residuals as a loss function - the same one that GB is using.

I also thought about lowering the learning rate or using only a subset of predictors for each iteration, but that could still be summed up to a single model representation eventually, so I guess it would bring no improvement.

What am I missing here? Is linear regression somehow inappropriate to use with Gradient Boosting? Is it because the linear regression uses the sum of squared residuals as a loss function? Are there any particular constraints on the weak predictors so they can be applied to Gradient Boosting?


  What am I missing here?


I don't think you're really missing anything!


  Another observation is that a sum of subsequent linear regression models can be represented as a single regression model as well (adding all intercepts and corresponding coefficients) so I cannot imagine how that could ever improve the model. The last observation is that a linear regression (the most typical approach) is using sum of squared residuals as a loss function - the same one that GB is using.


Seems to me that you nailed it right there, and gave a short sketch of a proof that linear regression just beats boosting linear regressions in this setting.

To be pedantic, both methods are attempting to solve the following optimization problem

$$ \hat \beta = \text{argmin}_\beta (y - X \beta)^t (y - X \beta) $$

Linear regression just observes that you can solve it directly, by finding the solution to the linear equation

$$ X^t X \beta = X^t y $$

This automatically gives you the best possible value of $\beta$ out of all possibilities.

Boosting, whether your weak classifier is a one variable or multi variable regression, gives you a sequence of coefficient vectors $\beta_1, \beta_2, \ldots$.  The final model prediction is, as you observe, a sum, and has the same functional form as the full linear regressor

$$ X \beta_1 + X \beta_2 + \cdots + X \beta_n = X (\beta_1 + \beta_2 + \cdots + \beta_n) $$

Each of these steps is chosen to further decrease the sum of squared errors.  But we could have found the minimum possible sum of square errors within this functional form by just performing a full linear regression to begin with.

A possible defense of boosting in this situation could be the implicit regularization it provides.  Possibly (I haven't played with this) you could use the early stopping feature of a gradient booster, along with a cross validation, to stop short of the full linear regression. This would provide a regularization to your regression, and possibly help with overfitting.  This is not particularly practical, as one has very efficient and well understood options like ridge regression and the elastic net in this setting.

Boosting shines when there is no terse functional form around.  Boosting decision trees lets the functional form of the regressor/classifier evolve slowly to fit the data, often resulting in complex shapes one could not have dreamed up by hand and eye.  When a simple functional form is desired, boosting is not going to help you find it (or at least is probably a rather inefficient way to find it).
