<p>The MSE is essentially a squared Euclidean distance between two vectors, say <span class="math-container">$\mathbf y$</span> and <span class="math-container">$\hat{\mathbf y}$</span>, where <span class="math-container">$\mathbf y$</span> is the actual economic data over <span class="math-container">$T$</span> periods and <span class="math-container">$\hat{\mathbf{y}}$</span> the predicted values. A natural extension of this to matrices <span class="math-container">$\mathbf Y=(y_{it})$</span> and <span class="math-container">$\widehat{\mathbf Y}=(\hat y_{it})$</span> where <span class="math-container">$i=1,\dots,n$</span> and <span class="math-container">$t=1,\dots,T$</span> (<span class="math-container">$n$</span> variables over <span class="math-container">$T$</span> periods) would be
<span class="math-container">\begin{equation}
\text{MSE}^*=\frac{1}{nT}\sum_{i=1}^n\sum_{t=1}^T(y_{it}-\hat y_{it})^2.
\end{equation}</span></p>
<p>But beware of the <a href="https://stats.stackexchange.com/q/99171/61402">pitfall</a>. Of course there are other <a href="https://en.wikipedia.org/wiki/Matrix_norm" rel="nofollow noreferrer">matrix norms</a> available as well.</p>

CC BY-SA 4.0
2021-03-26T04:19:28.257Z
Thanks for multivariate MSE info.

However, the "units" are different for each dimension (eg GDP in dollars, inflation in %, unemployment could be % or absolute). Doesn't that mean I need to somehow "normalise" the values first, to make MSE a good measure?

A simple example: if I measure GDP in $, model Y (great at inflation, ok at GDP) might have a lower overall MSE than model X (slightly better at GDP, much worse on inflation). But if I measure GDP in cents (or Yen, if in Japan), any gap in GDP becomes dominant, and model X suddenly looks *way* better than Y.

Isn't this a big issue?
Mich55
CC BY-SA 4.0
2021-03-26T17:07:10.877Z
@Mich55: Unit differences are indeed an issue that I overlooked. Normalization would seem sensible but it's not obvious how it should be done. A search in the model evaluation literature (e.g. [this paper](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.461.5460&rep=rep1&type=pdf)) may be helpful.
Herr K.
CC BY-SA 4.0
2021-03-25T06:24:35.177Z
2021-03-25T06:24:35.177Z
Herr K.
<p>I have several macroeconomic models (eg two DSGEs and a VAR). They each produce forecasts for GDP, inflation and unemployment, and I have data to test them on, going back years.</p>
<p><strong>How can I say which model has the &quot;best&quot; overall performance?</strong></p>
<p>I realise this is vague, so here's more detail:</p>
<p>Assume that model X performs better than model Y on GDP, but worse on inflation.</p>
<p>If I'm just looking at GDP, I could just ask something like &quot;which model has the smallest mean square error?&quot; (predicted GDP - actual GDP, squared)
Similarly, if I only care about inflation, I could do the same thing.</p>
<p>But I actually care about forecast accuracy over all three variables (GDP/inf/unemployment).</p>
<p><strong>Is there a way to assess accuracy over all three variables, without just assigning weights to each MSE value and adding them up?</strong></p>
<p>Thanks for your help!</p>

<econometrics><forecasting>
How good is a forecast? (Many outputs)
Mich55
# How good is a forecast? (Many outputs)

(Asked by: username_0 on 2021-03-25)

I have several macroeconomic models (eg two DSGEs and a VAR). They each produce forecasts for GDP, inflation and unemployment, and I have data to test them on, going back years.

**How can I say which model has the "best" overall performance?**

I realise this is vague, so here's more detail:

Assume that model X performs better than model Y on GDP, but worse on inflation.

If I'm just looking at GDP, I could just ask something like "which model has the smallest mean square error?" (predicted GDP - actual GDP, squared)
Similarly, if I only care about inflation, I could do the same thing.

But I actually care about forecast accuracy over all three variables (GDP/inf/unemployment).

**Is there a way to assess accuracy over all three variables, without just assigning weights to each MSE value and adding them up?**

Thanks for your help!

**Tags:** econometrics, forecasting


---

## 1 answers

### Answer by username_1 on 2021-03-25. Score: 2

The MSE is essentially a squared Euclidean distance between two vectors, say $\mathbf y$ and $\hat{\mathbf y}$, where $\mathbf y$ is the actual economic data over $T$ periods and $\hat{\mathbf{y}}$ the predicted values. A natural extension of this to matrices $\mathbf Y=(y\_{it})$ and $\widehat{\mathbf Y}=(\hat y\_{it})$ where $i=1,\dots,n$ and $t=1,\dots,T$ ($n$ variables over $T$ periods) would be
\begin{equation}
\text{MSE}^\*=\frac{1}{nT}\sum\_{i=1}^n\sum\_{t=1}^T(y\_{it}-\hat y\_{it})^2.
\end{equation}

But beware of the [pitfall](https://stats.stackexchange.com/q/99171/61402). Of course there are other [matrix norms](https://en.wikipedia.org/wiki/Matrix_norm) available as well.

> **Comments:**
> **username_0**: Thanks for multivariate MSE info.  However, the "units" are different for each dimension (eg GDP in dollars, inflation in %, unemployment could be % or absolute). Doesn't that mean I need to somehow "normalise" the values first, to make MSE a good measure?  A simple example: if I measure GDP in $, model Y (great at inflation, ok at GDP) might have a lower overall MSE than model X (slightly better at GDP, much worse on inflation). But if I measure GDP in cents (or Yen, if in Japan), any gap in GDP becomes dominant, and model X suddenly looks *way* better than Y.  Isn't this a big issue?  
> **username_1**: @username_0: Unit differences are indeed an issue that I overlooked. Normalization would seem sensible but it's not obvious how it should be done. A search in the model evaluation literature (e.g. [this paper](https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.461.5460&rep=rep1&type=pdf)) may be helpful.  

---

