<p>I think you might be getting terms mixed up. Let's follow the standard convention of <a href="https://en.wikipedia.org/wiki/Ensemble_Kalman_filter#Kalman_filter" rel="nofollow noreferrer">data assimilation</a> and call your model output from the (<span class="math-container">$k^{\text{th}}$</span> parameterization/forcing) <span class="math-container">$X_k$</span> and the observations <span class="math-container">$y$</span>. For simplicity's sake, let's just say that the observation operator is unitary (<span class="math-container">$H=I$</span>). The equation for the Root Mean Squared Error (RMSE) for the <span class="math-container">$k^{\text{th}}$</span> model (<span class="math-container">$k$</span> could be a unique index that is a combination of forcing and parameterization) is <span class="math-container">$$RMSE_k=\sqrt{\sum_i\frac{\left(x_{i,k}-y_i\right)^2}{N}}$$</span> where <span class="math-container">$x_i$</span> and <span class="math-container">$y_i$</span> are indices paired 'samples' from <span class="math-container">$X$</span> and <span class="math-container">$Y$</span> respectively. The sampling that you use performs RMSE sampled over time, but you can just as easily sample over space.</p>
<p>What is a metric that is easily comparable to RMSE that measures model variation? Let's follow what you suggested (more for the narrative). If we let the mean state for the <span class="math-container">$k^{\text{th}}$</span> model be <span class="math-container">$\bar{x}_k=\frac{1}{N}\sum_i x_{i,k}$</span> then the formula you suggested would actually be <span class="math-container">$$\sqrt{\sum_i\frac{\left(x_{i,k}-\bar{x}_k\right)^2}{N}}$$</span>. Now that looks a lot like the formula for the <a href="https://en.wikipedia.org/wiki/Standard_deviation" rel="nofollow noreferrer">standard deviation</a>. Rather, the way that we notated this formula shows the model-dependent standard deviation fluctuates on time. However, if we make a model mean variable <span class="math-container">$\hat{x}_i=\frac{1}{N}\sum_k x_{i,k}$</span>, then we can determine how much the different forcings and parameterizations cause the output to vary with time: <span class="math-container">$$\sigma_i=\sqrt{\sum_k\frac{\left(x_{i,k}-\hat{x}_i\right)^2}{N}}$$</span>.</p>
<p>So really, your formula should look like (reverting back to your notation): <span class="math-container">$$\frac{RMSE}{\sqrt{\frac{1}{N}(Y_{i,j}(t_k)-\bar{Y_j}(t_k))^2}}$$</span></p>
<p>Now, I think your idea is fine, but the interpretation is incorrect. You can legitimately have a number that is greater than one (for example, if you see no difference, the standard deviation is 0, therefore your metric reaches infinity). You also cannot say if the inclusion of the parameterization makes the model better based on this metric. That would require the examining the RMSE of each parameterization + forcing. Such an experiment shows relative roles that the parameterizations/forcings of similar kind have in creating the number of possible model outputs that could be causing model errors.</p>
<p>An example that I know of where such an analysis was conducted was <a href="https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2018MS001457" rel="nofollow noreferrer">Thomas et al. (2019)</a>. In it, the RMSE was computed and the model standard deviation was compared, with the standard deviation being smaller than the RMSE (therefore leading to numbers greater than 1, per the corrections [namely the square root] to your logic).</p>

CC BY-SA 4.0
2021-07-21T19:41:52.060Z
I should also note, it is best to treat a perturbation in the forcing as if it were a new parameterization. There shouldn't be a distinction between the two, until it comes to something like a sequential analysis or time analysis.
BarocliniCplusplus
CC BY-SA 4.0
2021-07-22T22:50:31.723Z
Thank you for the explanation and correction. Yes I forget the root square to compare with the RMSE to get a ratio homogeneous. In fact, I mix up the interpretation with this ratio with the same ratio by replacing each term by their mean over the forcing. 
$\frac{1}{N_f} \sum_j \frac{1}{N_t} \sum_k (Y_{ij}(t_k)-\bar{Y_j}(t_k))^2$ where $\bar{Y_j}(t_k)$ will be better to any set of observation. Thank you for the article
AnthonyB
CC BY-SA 4.0
2021-07-22T22:53:58.553Z
The problem with give a perturbation the forcing. It is difficult to keep a coherence between all the meteorogical variables and often the difference between the forcing is difficult to model it as perturbation.
AnthonyB
CC BY-SA 4.0
2021-07-21T19:39:51.187Z
2021-07-21T19:39:51.187Z
BarocliniCplusplus
<p>I want to validate a model and compare several parameterizations. I have some observations so I calculated the RMSE for each parameterization  with the best meteorological forcing.
However I want to nuance with the variability of the forcing. I suppose that more the forcing has an impact on outputs in its range of uncertainties, less the parameterization  explains the amelioration of the RMSE.</p>
<p>In my case, I have 5 forcing files and 2 different parameterizations, the output <span class="math-container">$Y$</span> is a time series. I name <span class="math-container">$Y_{ij}(t)$</span> each output with the forcing <span class="math-container">$i$</span> and the param <span class="math-container">$j$</span>
and I name the mean of <span class="math-container">$Y_{ij}(t)$</span> for each time step between all the forcing with the parameterization j as <span class="math-container">$\bar{Y_j}(t)$</span> and the time series of the observation is <span class="math-container">$y(t)$</span></p>
<p>I want to compare the RMSE with the following formulae (where I replace <span class="math-container">$y(t)$</span> by <span class="math-container">$\bar{Y_j}(t)$</span>):</p>
<p><span class="math-container">$$\frac{1}{N_t} \sum_k (Y_{ij}(t_k)-\bar{Y_j}(t_k))^2$$</span></p>
<p>Finally, I want to conclude that if the ratio  <span class="math-container">$ \frac{RMSE}{\sqrt{\frac{1}{N_t} \sum_k (Y_{ij}(t_k)-\bar{Y_j}(t_k))^2}} $</span> is near to 1 so it is difficult to conclude because the error on the forcing lead to the most of the sensitivity on the outputs and if it is near to 0, the parameterization  really improves the model to fit to the observations.</p>
<p>Is this approach relevant?</p>

<land-surface-models><validation>
Pertinence for a formulae similar of the RMSE to measure the meteorologic forcing impact on the variability on a model output
AnthonyB
# Pertinence for a formulae similar of the RMSE to measure the meteorologic forcing impact on the variability on a model output

(Asked by: username_0 on 2021-07-18)

I want to validate a model and compare several parameterizations. I have some observations so I calculated the RMSE for each parameterization with the best meteorological forcing.
However I want to nuance with the variability of the forcing. I suppose that more the forcing has an impact on outputs in its range of uncertainties, less the parameterization explains the amelioration of the RMSE.

In my case, I have 5 forcing files and 2 different parameterizations, the output $Y$ is a time series. I name $Y\_{ij}(t)$ each output with the forcing $i$ and the param $j$
and I name the mean of $Y\_{ij}(t)$ for each time step between all the forcing with the parameterization j as $\bar{Y\_j}(t)$ and the time series of the observation is $y(t)$

I want to compare the RMSE with the following formulae (where I replace $y(t)$ by $\bar{Y\_j}(t)$):

$$\frac{1}{N\_t} \sum\_k (Y\_{ij}(t\_k)-\bar{Y\_j}(t\_k))^2$$

Finally, I want to conclude that if the ratio $ \frac{RMSE}{\sqrt{\frac{1}{N\_t} \sum\_k (Y\_{ij}(t\_k)-\bar{Y\_j}(t\_k))^2}} $ is near to 1 so it is difficult to conclude because the error on the forcing lead to the most of the sensitivity on the outputs and if it is near to 0, the parameterization really improves the model to fit to the observations.

Is this approach relevant?

**Tags:** land-surface-models, validation


---

## 1 answers

### Answer by username_1 on 2021-07-21. Score: 2 | Accepted answer

I think you might be getting terms mixed up. Let's follow the standard convention of [data assimilation](https://en.wikipedia.org/wiki/Ensemble_Kalman_filter#Kalman_filter) and call your model output from the ($k^{\text{th}}$ parameterization/forcing) $X\_k$ and the observations $y$. For simplicity's sake, let's just say that the observation operator is unitary ($H=I$). The equation for the Root Mean Squared Error (RMSE) for the $k^{\text{th}}$ model ($k$ could be a unique index that is a combination of forcing and parameterization) is $$RMSE\_k=\sqrt{\sum\_i\frac{\left(x\_{i,k}-y\_i\right)^2}{N}}$$ where $x\_i$ and $y\_i$ are indices paired 'samples' from $X$ and $Y$ respectively. The sampling that you use performs RMSE sampled over time, but you can just as easily sample over space.

What is a metric that is easily comparable to RMSE that measures model variation? Let's follow what you suggested (more for the narrative). If we let the mean state for the $k^{\text{th}}$ model be $\bar{x}\_k=\frac{1}{N}\sum\_i x\_{i,k}$ then the formula you suggested would actually be $$\sqrt{\sum\_i\frac{\left(x\_{i,k}-\bar{x}\_k\right)^2}{N}}$$. Now that looks a lot like the formula for the [standard deviation](https://en.wikipedia.org/wiki/Standard_deviation). Rather, the way that we notated this formula shows the model-dependent standard deviation fluctuates on time. However, if we make a model mean variable $\hat{x}\_i=\frac{1}{N}\sum\_k x\_{i,k}$, then we can determine how much the different forcings and parameterizations cause the output to vary with time: $$\sigma\_i=\sqrt{\sum\_k\frac{\left(x\_{i,k}-\hat{x}\_i\right)^2}{N}}$$.

So really, your formula should look like (reverting back to your notation): $$\frac{RMSE}{\sqrt{\frac{1}{N}(Y\_{i,j}(t\_k)-\bar{Y\_j}(t\_k))^2}}$$

Now, I think your idea is fine, but the interpretation is incorrect. You can legitimately have a number that is greater than one (for example, if you see no difference, the standard deviation is 0, therefore your metric reaches infinity). You also cannot say if the inclusion of the parameterization makes the model better based on this metric. That would require the examining the RMSE of each parameterization + forcing. Such an experiment shows relative roles that the parameterizations/forcings of similar kind have in creating the number of possible model outputs that could be causing model errors.

An example that I know of where such an analysis was conducted was [Thomas et al. (2019)](https://agupubs.onlinelibrary.wiley.com/doi/full/10.1029/2018MS001457). In it, the RMSE was computed and the model standard deviation was compared, with the standard deviation being smaller than the RMSE (therefore leading to numbers greater than 1, per the corrections [namely the square root] to your logic).

> **Comments:**
> **username_1**: I should also note, it is best to treat a perturbation in the forcing as if it were a new parameterization. There shouldn't be a distinction between the two, until it comes to something like a sequential analysis or time analysis.  
> **username_0**: Thank you for the explanation and correction. Yes I forget the root square to compare with the RMSE to get a ratio homogeneous. In fact, I mix up the interpretation with this ratio with the same ratio by replacing each term by their mean over the forcing.  $\frac{1}{N_f} \sum_j \frac{1}{N_t} \sum_k (Y_{ij}(t_k)-\bar{Y_j}(t_k))^2$ where $\bar{Y_j}(t_k)$ will be better to any set of observation. Thank you for the article  
> **username_0**: The problem with give a perturbation the forcing. It is difficult to keep a coherence between all the meteorogical variables and often the difference between the forcing is difficult to model it as perturbation.  

---

