<p><strong>Question:</strong> I want to clarify my understanding of the basics of OLS regression in matrix form.</p>
<p>Let's assume we have 2 different independent variables <span class="math-container">$x_1$</span> and <span class="math-container">$x_2$</span>.</p>
<p>Our 'model' will be the plane that lives in <span class="math-container">$\mathbb{R^3}$</span> that minimises the sum of squared distances between each point on the plane corresponding to observations of our pair of independent variable points <span class="math-container">$x_{1i}$</span> &amp; <span class="math-container">$x_{2i}$</span> and the corresponding point <span class="math-container">$y_i$</span>. These individual distance are our estimated residuals call them <span class="math-container">$\bar μ_i$</span></p>
<p>Vector form: In vector form we instead have single vectors in <span class="math-container">$n$</span> dimensional space, whereby our number of observations defines <span class="math-container">$n$</span>.
<span class="math-container">$y$</span> and each <span class="math-container">$x_i$</span> variable form <span class="math-container">$n$</span> dimensional vectors in this space.</p>
<p>The span of these <span class="math-container">$x_i$</span> vectors will form a hyperplane <span class="math-container">$X$</span> in this <span class="math-container">$n$</span> dimensional space. Now our vector <span class="math-container">$\hat y$</span> which represents our model. This is the orthogonal projection of <span class="math-container">$y$</span> onto the plane <span class="math-container">$X$</span>, and the coefficients that define the linear combination of <span class="math-container">$x_i$</span> vectors that give <span class="math-container">$\hat y$</span> are the weights of our <span class="math-container">$β_ι$</span> parameters in our model.</p>
<p>What's bothering me is that in <a href="https://youtu.be/oWuhZuLOEFY?t=206" rel="nofollow noreferrer">this lecture series</a> the author states that.</p>
<blockquote>
<p>In OLS regression we are trying to get as close to this dependent variable vector <span class="math-container">$y$</span> as we can, given that we don't have a vector or a space <span class="math-container">$X$</span> which is as highly dimensional [as <span class="math-container">$y$</span>].</p>
</blockquote>
<p>What's bothering me here is that, each of the vectors <span class="math-container">$x_i$</span> are as &quot;highly dimensional&quot; as <span class="math-container">$y$</span> because they live in the same <span class="math-container">$n$</span> dimensional space, but i appreciate that <span class="math-container">$X$</span> does not span all of <span class="math-container">$\mathbb{R^n}$</span> and so we can not get y with just a linear combination of our different <span class="math-container">$x_i$</span>. <strong>Is this interpretation correct?</strong></p>
<p><strong>Question:</strong> If we had the &quot;true model&quot;  then this distance between the true plane call it <span class="math-container">$\bar X$</span> and <span class="math-container">$y$</span> would be our error term vector <span class="math-container">$u$</span> and i assume that <span class="math-container">$u$</span> would itself be the linear combination or many other  unaccounted for independent variables?</p>
<p><strong>Question:</strong> When i try and visualise this, naturally all i come up with is a plane in <span class="math-container">$\mathbb{R^3}$</span> and then <span class="math-container">$y$</span> is outside of this plane. This visual analogy implies that only one <span class="math-container">$x$</span> variable is missing i.e. we just need a 3rd linearly independent vector. But surely for any dimension of space we chose. Our <span class="math-container">$y$</span> is just one vector away from <span class="math-container">$X$</span> as we can always point wise define an extra vector <span class="math-container">$y - lin(X)$</span> and indeed if we take <span class="math-container">$\hat y$</span> to be our vector <span class="math-container">$\in X$</span> then <span class="math-container">$y - \hat y$</span> is just <span class="math-container">$\bar u$</span> our residual vector.</p>
<p>So i'm unclear how to think about this missing extra dimensionality the author mentions. I suppose in reality this <span class="math-container">$\bar u$</span> could be decomposed into many more LI vectors of uncounted for independent variables. But I would appreciate clarification.</p>

CC BY-SA 4.0
2023-06-19T23:03:25.530Z
How does differ from your [other question](https://stats.stackexchange.com/q/619186/247274) that has already attracted some useful comments?
Dave
CC BY-SA 4.0
2023-06-21T08:34:52.630Z
Hi Dave thanks for commenting. Yes they are almost identical. The reason for the duel post is that. 1) I first posted here as it's econometrics, but then thought maybe it's more appropriate for statistics. 2) I have posted in more than one stack forum before and got different answers depending on the community. Economists vs statasticians vs mathmeticans sometimes provide different answers focusses etc. So it's quite helpful getting both. i appreciate this is maybe problematic for the "knowledge base" gaol of stack, as duplicates are undesirable, but from a user perspective it is helpful!
CormJack
CC BY-SA 4.0
2023-06-21T08:35:45.847Z
@Dave If you have any time i'd really appreciate your thoughts on this question. Similar topic, but different focus. https://economics.stackexchange.com/questions/55700/vector-visualisation-of-regression-model-vs-scatter-plot-style
CormJack
<econometrics><statistics><linear-algebra>
Intuition of Linear Algebra visualisation for regression
CormJack
# Intuition of Linear Algebra visualisation for regression

(Asked by: username_0 on 2023-06-19)

**Question:** I want to clarify my understanding of the basics of OLS regression in matrix form.

Let's assume we have 2 different independent variables $x\_1$ and $x\_2$.

Our 'model' will be the plane that lives in $\mathbb{R^3}$ that minimises the sum of squared distances between each point on the plane corresponding to observations of our pair of independent variable points $x\_{1i}$ & $x\_{2i}$ and the corresponding point $y\_i$. These individual distance are our estimated residuals call them $\bar μ\_i$

Vector form: In vector form we instead have single vectors in $n$ dimensional space, whereby our number of observations defines $n$.
$y$ and each $x\_i$ variable form $n$ dimensional vectors in this space.

The span of these $x\_i$ vectors will form a hyperplane $X$ in this $n$ dimensional space. Now our vector $\hat y$ which represents our model. This is the orthogonal projection of $y$ onto the plane $X$, and the coefficients that define the linear combination of $x\_i$ vectors that give $\hat y$ are the weights of our $β\_ι$ parameters in our model.

What's bothering me is that in [this lecture series](https://youtu.be/oWuhZuLOEFY?t=206) the author states that.

> In OLS regression we are trying to get as close to this dependent variable vector $y$ as we can, given that we don't have a vector or a space $X$ which is as highly dimensional [as $y$].

What's bothering me here is that, each of the vectors $x\_i$ are as "highly dimensional" as $y$ because they live in the same $n$ dimensional space, but i appreciate that $X$ does not span all of $\mathbb{R^n}$ and so we can not get y with just a linear combination of our different $x\_i$. **Is this interpretation correct?**

**Question:** If we had the "true model" then this distance between the true plane call it $\bar X$ and $y$ would be our error term vector $u$ and i assume that $u$ would itself be the linear combination or many other unaccounted for independent variables?

**Question:** When i try and visualise this, naturally all i come up with is a plane in $\mathbb{R^3}$ and then $y$ is outside of this plane. This visual analogy implies that only one $x$ variable is missing i.e. we just need a 3rd linearly independent vector. But surely for any dimension of space we chose. Our $y$ is just one vector away from $X$ as we can always point wise define an extra vector $y - lin(X)$ and indeed if we take $\hat y$ to be our vector $\in X$ then $y - \hat y$ is just $\bar u$ our residual vector.

So i'm unclear how to think about this missing extra dimensionality the author mentions. I suppose in reality this $\bar u$ could be decomposed into many more LI vectors of uncounted for independent variables. But I would appreciate clarification.

**Tags:** econometrics, statistics, linear-algebra

> **Comments:**
> **username_1**: How does differ from your [other question](https://stats.stackexchange.com/q/619186/247274) that has already attracted some useful comments?  
> **username_0**: Hi username_1 thanks for commenting. Yes they are almost identical. The reason for the duel post is that. 1) I first posted here as it's econometrics, but then thought maybe it's more appropriate for statistics. 2) I have posted in more than one stack forum before and got different answers depending on the community. Economists vs statasticians vs mathmeticans sometimes provide different answers focusses etc. So it's quite helpful getting both. i appreciate this is maybe problematic for the "knowledge base" gaol of stack, as duplicates are undesirable, but from a user perspective it is helpful!  
> **username_0**: @username_1 If you have any time i'd really appreciate your thoughts on this question. Similar topic, but different focus. https://economics.stackexchange.com/questions/55700/vector-visualisation-of-regression-model-vs-scatter-plot-style  

---

## 0 answers

