<p>For prediction, yes you can consider the models
$$
y = \beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p + \gamma_2 \hat{y}^2 + \cdots + \gamma_m \hat{y}^m + error,
$$
where $\hat{y}$ represents the first-step OLS fitted values and $m$ is chosen by something like cross validation.</p>

<p>I haven't seen this approach used before. My personal guess is that this approach is not as useful as other commonly used ones (SVM, splines, GAM, etc.). For example, if $p$ is large (in comparison to the number of observations $n$), the first-step OLS may already be overfitting so including $\hat{y}$ is not practical. (Yes, you can use lasso residuals but that's a different story.) If $p$ is small, nonlinearity can perhaps be better handled by splines or even by simply augmenting the equation with quadratic and cubic terms of the features. Some generalized additive models (GAM) are already there too.</p>

<p>My personal experience is that nonlinearity is not so important (for prediction using economic data). It is usually a lot more important to avoid overfitting nicely. To me, your suggestion seems to be useful in some cases but not in many.</p>

<p>That said, I do not want to dissuade you from pursuing this issue, although there is a (high) chance of ending up with the conclusion that it is not very useful given the availability of other methods. BTW, you would already know it, but just in case, Hastie, Tibshirani and Friedman's book (The Elements of Statistical Learning) is helpful.</p>

CC BY-SA 4.0
2018-05-29T02:42:13.813Z
Hi: you may also want to look at discussions of  the  autoregressive distributed lag or the koyck distributed lag. These are not terribly re econometric approaches not terribly related to what you suggest  and they  won't  talk about why your approach ( I think you're approach is to fit as many powers as possible which is prone to overfitting )  is not necessarily useful but you may find  them  interesting as approaches on their own. they are both time-series modelling approaches so, if you're problem is not time series related, they won't be helpful.
mark leeds
CC BY-SA 4.0
2019-08-21T13:42:28.313Z
@markleeds Regarding your concern of overfitting, the answer mentions that the degree of the polynomial (model selection) is performed via something like cross-validation. Cross-validation exists precisely to avoid overfitting. Most econometrics textbooks don't use cross validation, because their primary concern is inference rather than prediction.
jmbejara
CC BY-SA 4.0
2019-08-21T18:28:35.483Z
@jmbejara. good point about cross-validation. thanks.
mark leeds
CC BY-SA 4.0
2018-05-27T13:58:41.563Z
2018-05-27T13:58:41.563Z
chan1142
<p>If my goal were prediction (e.g. of propensity scores), why couldn't I control for higher order terms of the model equation? For example, why not estimate the model and then control for squares and cubes of the predicted values in a second stage and get an even better estimate of the correct predicted values (rather than just a test of functional form)? I know the standard errors would be wrong but couldn't I bootstrap the process?</p>

CC BY-SA 4.0
2018-05-25T16:06:33.390Z
Why stop at cubes? Shouldn't you control for all powers all the way to the 99th?
Giskard
CC BY-SA 4.0
2018-05-25T17:06:40.680Z
There is a difference between using a flexible functional form and overfitting. With regards to my question, let's assume I'm not overfitting the model.
Kris B.
CC BY-SA 4.0
2018-05-25T19:16:46.457Z
Hi:  Since, a regression is just minimizing a sum of squares between the value of the function used ( whatever functional form you like ) and the response,  you can use whatever functional form you like but it doesn't mean the results are useful. I'm not clear on what you mean by "control for squares and cubes of predicted values" ????
mark leeds
CC BY-SA 4.0
2018-05-25T19:57:20.153Z
Who said you can't?
Alecos Papadopoulos
CC BY-SA 4.0
2018-05-25T20:58:36.517Z
Nobody did. I guess I just thought you couldn't because I've never seen it done.
Kris B.
CC BY-SA 4.0
2018-05-27T15:50:34.523Z
You can. The crux here is how you justify controlling for $x^2$ but not $x^3$, $x^4$, ...,$x^N$. You get into really arbitrary waters. Further, doing stuff like this will likely give you better in-sample fit but terrible out-of-sample predictions. It is an example of overfitting.  So...you can...but within reason and with good justification.
123
<econometrics><applied-econometrics>
Why can't we control for squares and quadratics of predicted values in regression?
Kris B.
# Why can't we control for squares and quadratics of predicted values in regression?

(Asked by: username_0 on 2018-05-25)

If my goal were prediction (e.g. of propensity scores), why couldn't I control for higher order terms of the model equation? For example, why not estimate the model and then control for squares and cubes of the predicted values in a second stage and get an even better estimate of the correct predicted values (rather than just a test of functional form)? I know the standard errors would be wrong but couldn't I bootstrap the process?

**Tags:** econometrics, applied-econometrics

> **Comments:**
> **username_1**: Why stop at cubes? Shouldn't you control for all powers all the way to the 99th?  
> **username_0**: There is a difference between using a flexible functional form and overfitting. With regards to my question, let's assume I'm not overfitting the model.  
> **username_2**: Hi:  Since, a regression is just minimizing a sum of squares between the value of the function used ( whatever functional form you like ) and the response,  you can use whatever functional form you like but it doesn't mean the results are useful. I'm not clear on what you mean by "control for squares and cubes of predicted values" ????  
> **username_3**: Who said you can't?  
> **username_0**: Nobody did. I guess I just thought you couldn't because I've never seen it done.  
> **username_4**: You can. The crux here is how you justify controlling for $x^2$ but not $x^3$, $x^4$, ...,$x^N$. You get into really arbitrary waters. Further, doing stuff like this will likely give you better in-sample fit but terrible out-of-sample predictions. It is an example of overfitting.  So...you can...but within reason and with good justification.  

---

## 1 answers

### Answer by username_5 on 2018-05-27. Score: 1

For prediction, yes you can consider the models
$$
y = \beta\_0 + \beta\_1 x\_1 + \cdots + \beta\_p x\_p + \gamma\_2 \hat{y}^2 + \cdots + \gamma\_m \hat{y}^m + error,
$$
where $\hat{y}$ represents the first-step OLS fitted values and $m$ is chosen by something like cross validation.

I haven't seen this approach used before. My personal guess is that this approach is not as useful as other commonly used ones (SVM, splines, GAM, etc.). For example, if $p$ is large (in comparison to the number of observations $n$), the first-step OLS may already be overfitting so including $\hat{y}$ is not practical. (Yes, you can use lasso residuals but that's a different story.) If $p$ is small, nonlinearity can perhaps be better handled by splines or even by simply augmenting the equation with quadratic and cubic terms of the features. Some generalized additive models (GAM) are already there too.

My personal experience is that nonlinearity is not so important (for prediction using economic data). It is usually a lot more important to avoid overfitting nicely. To me, your suggestion seems to be useful in some cases but not in many.

That said, I do not want to dissuade you from pursuing this issue, although there is a (high) chance of ending up with the conclusion that it is not very useful given the availability of other methods. BTW, you would already know it, but just in case, Hastie, Tibshirani and Friedman's book (The Elements of Statistical Learning) is helpful.

> **Comments:**
> **username_2**: Hi: you may also want to look at discussions of  the  autoregressive distributed lag or the koyck distributed lag. These are not terribly re econometric approaches not terribly related to what you suggest  and they  won't  talk about why your approach ( I think you're approach is to fit as many powers as possible which is prone to overfitting )  is not necessarily useful but you may find  them  interesting as approaches on their own. they are both time-series modelling approaches so, if you're problem is not time series related, they won't be helpful.  
> **username_6**: @markleeds Regarding your concern of overfitting, the answer mentions that the degree of the polynomial (model selection) is performed via something like cross-validation. Cross-validation exists precisely to avoid overfitting. Most econometrics textbooks don't use cross validation, because their primary concern is inference rather than prediction.  
> **username_2**: @username_6. good point about cross-validation. thanks.  

---

