<p>I would like to show you some intuition behind it.</p>
<p>Well, first think about what does the regression exactly do?</p>
<p>Figuratively speaking, it filters out the variability (<em>&quot;information&quot;</em>) from the random component of some random variable. It does it by finding such object (e.g. line, plane, hyperplane, etc...) which <strong>minimizes the sum of squared residuals</strong> , which are a distance between dependent variable and this object, measured exactly in units of the dependent variable. In regression, this object is defined parametrically.</p>
<p>Now, what does that mean? You have your parameter <span class="math-container">$\hat{\beta_i}$</span> which is a random variable, meaning that by selecting a little different data, the result would be different. But here's <strong>the catch</strong>... For each <em>realization</em> of <span class="math-container">$\hat{\beta_i}$</span> you would have a <strong>set of realizations</strong> of residuals <span class="math-container">$\hat{e}$</span>. If you take new data from some population, you change only <strong>1</strong> value of <span class="math-container">$\hat{\beta_i}$</span> but as many values of residuals as the number of data you use. Thus if you change the data you change the residuals and the parameter! Are they therefore correlated because of coming from the same generation process? Well, not exactly...</p>
<p>Illustration: Consider the setup in which you have simple regression model <span class="math-container">$y = \beta_0 + \epsilon$</span> and you repeatedly take the sample of <strong>2</strong> data from population containing <span class="math-container">$n$</span>. And on each sample you perform the regression. See below:</p>
<p><a href="https://i.sstatic.net/ZpOK7.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ZpOK7.png" alt="independence" /></a></p>
<p>As you can see, for two arbitrarily given parameters <span class="math-container">$\hat{\beta_i}$</span> you can get similar residuals (see two pairs of red dots or two pairs of blue dots). They change <strong>randomly</strong> as the parameter changes. Thus <span class="math-container">$\hat{\beta_i}$</span> and <span class="math-container">$\hat{e}$</span> are independent</p>

CC BY-SA 4.0
2022-10-20T12:28:11.653Z
2022-10-20T12:28:11.653Z
Athaeneus
<p>If all assumptions hold (linear model, mean-zero error, homoskedasticity, no serial correlation and normal distribution), will the covariance between <span class="math-container">$\epsilon_{n+1}$</span>
and <span class="math-container">$\hat{\beta}_{n}$</span> (the estimator) be zero?</p>
<p>I know that the <span class="math-container">$cov(x_{i},\epsilon_{i})=0$</span> by the mean-zero error assumption and law of iterated expectations. And I think that the error term and the estimator need to be uncorrelated as well. Can somebody explain me why or maybe derive it mathematically?</p>
<p>Thank you in advance</p>

CC BY-SA 4.0
2022-10-19T10:19:16.717Z
What is the formulae for $\widehat{\beta}_n$? Why is $\widehat{\beta}_n$ random? How does this influence the covariance?
Bertrand
CC BY-SA 4.0
2022-12-09T09:32:09.927Z
See also: https://stats.stackexchange.com/questions/595756/check-covx-e-0-using-residuals
Papayapap
<regression>
Correlation between errors and estimates
dewewdew
# Correlation between errors and estimates

(Asked by: username_0 on 2022-10-19)

If all assumptions hold (linear model, mean-zero error, homoskedasticity, no serial correlation and normal distribution), will the covariance between $\epsilon\_{n+1}$
and $\hat{\beta}\_{n}$ (the estimator) be zero?

I know that the $cov(x\_{i},\epsilon\_{i})=0$ by the mean-zero error assumption and law of iterated expectations. And I think that the error term and the estimator need to be uncorrelated as well. Can somebody explain me why or maybe derive it mathematically?

Thank you in advance

**Tags:** regression

> **Comments:**
> **username_1**: What is the formulae for $\widehat{\beta}_n$? Why is $\widehat{\beta}_n$ random? How does this influence the covariance?  
> **username_2**: See also: https://stats.stackexchange.com/questions/595756/check-covx-e-0-using-residuals  

---

## 1 answers

### Answer by username_3 on 2022-10-20. Score: 2 | Accepted answer

I would like to show you some intuition behind it.

Well, first think about what does the regression exactly do?

Figuratively speaking, it filters out the variability (*"information"*) from the random component of some random variable. It does it by finding such object (e.g. line, plane, hyperplane, etc...) which **minimizes the sum of squared residuals** , which are a distance between dependent variable and this object, measured exactly in units of the dependent variable. In regression, this object is defined parametrically.

Now, what does that mean? You have your parameter $\hat{\beta\_i}$ which is a random variable, meaning that by selecting a little different data, the result would be different. But here's **the catch**... For each *realization* of $\hat{\beta\_i}$ you would have a **set of realizations** of residuals $\hat{e}$. If you take new data from some population, you change only **1** value of $\hat{\beta\_i}$ but as many values of residuals as the number of data you use. Thus if you change the data you change the residuals and the parameter! Are they therefore correlated because of coming from the same generation process? Well, not exactly...

Illustration: Consider the setup in which you have simple regression model $y = \beta\_0 + \epsilon$ and you repeatedly take the sample of **2** data from population containing $n$. And on each sample you perform the regression. See below:

[![independence](https://i.sstatic.net/ZpOK7.png)](https://i.sstatic.net/ZpOK7.png)

As you can see, for two arbitrarily given parameters $\hat{\beta\_i}$ you can get similar residuals (see two pairs of red dots or two pairs of blue dots). They change **randomly** as the parameter changes. Thus $\hat{\beta\_i}$ and $\hat{e}$ are independent


---

