<p>How many observations do you have for each household? If it's a small number, then the household FE won't be estimated well. This will &quot;average out&quot; across households, and the estimate of the effect of years of schooling won't be biased because of it, but if you predict out of sample the imprecise and volatile HH FE point estimates will cause poor prediction out of sample.</p>

CC BY-SA 4.0
2022-02-23T09:16:59.657Z
2022-02-23T09:16:59.657Z
Michael Gmeiner
<p>To test whether it is appropriate to even use certain types of fixed effects, would it ever be appropriate to use a cross-validation methodology? Let's say that I have this type of mental model:</p>
<p><span class="math-container">$$\text{wage} = \text{years of schooling} + \text{household FE} + \text{industry FE} + \text{year FE}$$</span></p>
<p>Suppose, however, that I use cross-validation to test this time series model out of sample, and somehow find out that if I <em><strong>don't</strong></em> include household FE, that my predictions of wage are more accurate (as measured by, say, lower out of sample RMSE). <em>In that case</em>, would it be more appropriate to use this model instead?</p>
<p><span class="math-container">$$\text{wage} = \text{years of schooling} + \text{industry FE} + \text{year FE}$$</span></p>
<p>Why or why not?</p>
<p>My own intuition suggests that if there <em><strong>truly is</strong></em> household-level effects that are confounding estimation, then adding household FE should increase the accuracy of out-of-sample predictions. At the same time, those predictions seem to rely on accurate estimation of each household's fixed effects, which may not be accurate for an individual household. On the other hand, I know that one would never naively throw variables to predict an outcome to get a causal model.</p>
<p>As another way of putting it, are the following statements true or false:</p>
<p>'if there truly is a causal relationship with one variable, then adding that variable should help improve prediction.'</p>
<p>'if there is a causal relationship, AND my prediction is <em>more inaccurate</em> when using those specific causal variables, then there is no way I can accurately identify the causal parameter of interest' (in this case the slope on years of schooling)</p>

CC BY-SA 4.0
2022-02-23T11:38:50.653Z
Why do you want to **predict** wage? This sounds like data science but not economics.
Alalalalaki
CC BY-SA 4.0
2022-02-23T14:47:22.457Z
I guess my question is more in line with: 'if there is truly a causal relationship, then it should help improve prediction.'
Daycent
CC BY-SA 4.0
2022-02-23T14:58:34.587Z
What casual relationship here? Some unknown characteristics varying across households affect wage?
Alalalalaki
CC BY-SA 4.0
2022-02-23T15:02:24.397Z
yes, (e.g. household network, household language, etc.)
Daycent
<econometrics><applied-econometrics><statistics><fixed-effects>
Cross validation as a means of validating fixed effects?
Daycent
# Cross validation as a means of validating fixed effects?

(Asked by: username_0 on 2022-02-22)

To test whether it is appropriate to even use certain types of fixed effects, would it ever be appropriate to use a cross-validation methodology? Let's say that I have this type of mental model:

$$\text{wage} = \text{years of schooling} + \text{household FE} + \text{industry FE} + \text{year FE}$$

Suppose, however, that I use cross-validation to test this time series model out of sample, and somehow find out that if I ***don't*** include household FE, that my predictions of wage are more accurate (as measured by, say, lower out of sample RMSE). *In that case*, would it be more appropriate to use this model instead?

$$\text{wage} = \text{years of schooling} + \text{industry FE} + \text{year FE}$$

Why or why not?

My own intuition suggests that if there ***truly is*** household-level effects that are confounding estimation, then adding household FE should increase the accuracy of out-of-sample predictions. At the same time, those predictions seem to rely on accurate estimation of each household's fixed effects, which may not be accurate for an individual household. On the other hand, I know that one would never naively throw variables to predict an outcome to get a causal model.

As another way of putting it, are the following statements true or false:

'if there truly is a causal relationship with one variable, then adding that variable should help improve prediction.'

'if there is a causal relationship, AND my prediction is *more inaccurate* when using those specific causal variables, then there is no way I can accurately identify the causal parameter of interest' (in this case the slope on years of schooling)

**Tags:** econometrics, applied-econometrics, statistics, fixed-effects

> **Comments:**
> **username_1**: Why do you want to **predict** wage? This sounds like data science but not economics.  
> **username_0**: I guess my question is more in line with: 'if there is truly a causal relationship, then it should help improve prediction.'  
> **username_1**: What casual relationship here? Some unknown characteristics varying across households affect wage?  
> **username_0**: yes, (e.g. household network, household language, etc.)  

---

## 1 answers

### Answer by username_2 on 2022-02-23. Score: 2 | Accepted answer

How many observations do you have for each household? If it's a small number, then the household FE won't be estimated well. This will "average out" across households, and the estimate of the effect of years of schooling won't be biased because of it, but if you predict out of sample the imprecise and volatile HH FE point estimates will cause poor prediction out of sample.


---

