<p>The late George Box famously said:</p>

<blockquote>
  <p>All models are wrong. Some models are useful</p>
</blockquote>

<p>All models are wrong.</p>

<p>That's important.</p>

<p><em>All</em> models are wrong.</p>

<p>In any real-world system that we're interested in, the only accurate model of the system, is the system itself. Anything else, any simplification, gives wrong answers.</p>

<p>Some models are useful.</p>

<p>That's important, too.</p>

<p>Some models help us structure our discussions of the uncertainties (H/T Prof Neil Strachan)</p>

<p>Some models allow us to test "what if" scenarios, and look at what the relative changes might be. Now, in the real world, exogenous influences that the model can't account for, would mean that the actual outcomes would be different: but if the model gives a near-enough account of the scale of relative changes, we can still make informed judgements on the basis of the models. Even when it involves pesky things such as turbulence.</p>

<p>Some models give us insights into emergent properties of systems: a lot of the really interesting problems are emergent properties that aren't necessarily obvious from just looking at the basic rules of the system, and any insights we can gain are valuable.</p>

<p>Some models allow us to do experiments on systems that we couldn't possibly experiment on. And although we know the models are wrong, they can at least give us an indication of the range of possible outcomes.</p>

<p>That's not to defend the use of bad models to do bad work. There's far too much of that in my field, and I do not like it. As Simon W says, it is possible to test models to see if they are suitable for a particular purpose. To continue using a model <em>for a purpose where it is known to be inapplicable</em>, is charlatanry, not academia.</p>

CC BY-SA 3.0
2015-01-08T06:29:12.737Z
All models are wrong, except, of course, [the big one](http://hitchhikers.wikia.com/wiki/Earth).
naught101
CC BY-SA 3.0
2015-01-06T00:39:46.287Z
2015-01-06T14:25:32.600Z
2015-01-06T14:25:32.600Z
410 gone
<p>Great question, with some interesting answers. It's obviously struck a chord around here.</p>

<p>I'd just like to add the abstract from one of my favourite papers, by Oreskes, Shrader-Frechette, &amp; Beiitz (1994):</p>

<blockquote>
  <p>Verification and validation of numerical models of natural systems is impossible. This is because natural systems are never closed and because model results are always non-unique. Models can be confirmed by the demonstration of agreement between observation and prediction, but confirmation is inherently partial. Complete confirmation is logically precluded by the fallacy of affirming the consequent and by incomplete access to natural phenomena. Models can only be evaluated in relative terms, and their predictive value is always open to question. The primary value of models is heuristic.</p>
</blockquote>

<p>Basically, in a complex system, you'd never expect a model's results to match observed data - there is noise everywhere, but also these systems contain deterministic chaos. You can simulate the behaviour of such a system, but you can't simulate the <em>results</em>. Even if your model is exactly correct, with the correct inputs, you won't get the same behaviour, because even the slightest value truncation will lead to a significant divergence, eventually.</p>

<p>Also, even if you do get the <em>exact result</em> you can't know that you got the right result for the right reasons - equifinality is a real problem. <em>Especially</em> in a model with more than two interacting calibrated parameters. There is no guarantee that the calibration process will give you better parameters, you can only be sure that it will give you better results <em>for the calibration data</em>. Even calibration using a cross-validation methodology (train-and-test) with actually independent data (rare) might fail, because the validation part can fail for the above reasons.</p>

<p>Anyway, everyone interested in that this topic should read that paper, I'm sure I've missed points here. It is a bit heavy-going, but that's because of the concepts - the paper is actually pretty enjoyable to read.</p>

<h3>References</h3>

<ul>
<li>Oreskes, N., Shrader-Frechette, K. &amp; Beiitz, K., 1994. Verification, validation, and confirmation of numerical models in the Earth sciences. Science, 263(5147), pp.641–646. Available at: <a href="http://search.proquest.com/docview/733194143/14125288294380DFD60/1?accountid=12763" rel="nofollow">ProQuest</a>.</li>
</ul>

CC BY-SA 3.0
2015-01-08T06:58:12.737Z
2016-10-14T03:08:34.700Z
2016-10-14T03:08:34.700Z
naught101
<p>A slightly different slant:</p>

<p>As energynumbers has pointed out, all models are wrong in one way or another. A fully accurate model of reality would be as complex as reality, so some simplification is always needed. This means that there cannot be any useful model whose predictions in every way are perfectly accurate (except possibly by accident).</p>

<p>Validating a model, therefore, is not a process for saying "Is this model correct" with a yes/no answer. Rather, it should be about,</p>

<ol>
<li>Deciding criteria the model must fulfill to be useful <em>for a given purpose</em>.</li>
<li>Assessing whether it fulfills those criteria.</li>
</ol>

<p><strong>If the model fails validation for one application, it may still be useful for another.</strong></p>

<p>Some examples of different validation criteria, taking a regional ocean model as an example:</p>

<p>One obvious choice is whether the predictions of current speeds or of water levels is most important. Even after selecting one of these, the way in which the predictions are assessed may vary. A reasonable default might be to assess the fit at every time point between measured and observed water levels in a number of places, and accept the model if measurements such as bias and RMSE are below defined thresholds. However, here are two examples of specific scenarios that justify different approaches:</p>

<ul>
<li><p>NOAA runs models whose primary application is the production of depth information for navigational aids. The critical statistic for them is not the general accuracy of their models, but the frequency with which the model overpredicts the water level - as this form of error could result in vessels running aground. [1]</p></li>
<li><p>Vested et al (1995) [2] give an example of a storm surge model that would provide flood warnings. It was tested not for the accuracy of all of its water level predictions, but for the accuracy of its predictions of <em>peak</em> water levels, as those are what would matter in operation.</p></li>
</ul>

<p>[1] NOAA <a href="http://www.nauticalcharts.noaa.gov/csdl/docs/RD_standards_Hess_etal.pdf" rel="nofollow">NOS STANDARDS FOR EVALUATING OPERATIONAL NOWCAST AND FORECAST HYDRODYNAMIC MODEL SYSTEMS Silver Spring, Maryland October 2003 noaa National Oceanic and Atmospheric Administration U.S.</a>,  NOAA Technical Report NOS CS 17, Oct. 2003</p>

<p>[2] H. J. Vested, J. W. Nielsen, H. R. Jensen, and K. B. Kristensen, “Skill Assessment of an Operational Hydrodynamic Forecast System for the North Sea and Danish Belts,” in Quantitative Skill Assessment for Coastal Ocean Models, vol. 47, D. R. Lynch and A. M. Davies, Eds. Washington DC: American Geophysical Union, 1995, pp. 373–396.</p>

CC BY-SA 3.0
2015-01-06T13:41:14.310Z
2016-10-13T13:26:07.090Z
2016-10-13T13:26:07.090Z
Semidiurnal Simon
<p>Since there are already two answers discussing possible merits in further analysis let me just add the other side of the coin; it could also be prudent to see the simulation as a failure and not analyse it any further.</p>

<p>That could be the case if it's state of the art that similar simulations reproduce these experimental values you were mentioning and if these simulation can reproduce the values you want to look at as well. 
In that case, it only makes sense to try to find out why you can't reproduce your experimental test values. If you do not want to spend much time in developing the model, then drop it altogether; no one will trust your results.</p>

<p>Trust is, in my opinion, the crucial point here. As mentioned in the previous answers it is not uncommon for simulations to give only (poor) estimates of experimental results. But it is all important in these cases that you can scrutinize why you still trust the results you want to get out of it. </p>

<p>To be honest, I do read your question that way that you have no good reason to be confident in further results. If that is the case my advice would be to step back and first think about whether you can trust them. To do that no general answer will help you; it depends too strongly on your system and what the state of the art is for these kind of simulations.</p>

CC BY-SA 3.0
2015-01-06T09:02:26.853Z
2015-01-06T09:02:26.853Z
taupunkt
<p>Yes there is value.  </p>

<p>1)Often times we use models to check our own understanding of a process.  If the model is built with all known factors included, it might still fail to produce results that can be verified with measurements. When the model is unable to reproduce a phenomenon, the test case can be used to assess what parts of the model need further development.</p>

<p>2)Sometimes systems are too complicated to model accurately (or perhaps there are too many unknowns in the initial or boundary conditions).  Although the model might fail to reproduce absolute values that are measured, it might still be able to produce realistic dynamic variance.  So, the model is still useful for studying the relationships between the dynamic processes that are involved.  This is sometimes how we find secondary effects that are not typically direct outcomes of the model processes.</p>

CC BY-SA 3.0
2015-01-05T23:42:55.903Z
2015-01-05T23:42:55.903Z
f.thorpe
<p>The answer depends very much on where the model fails. If it is very accurate in some regimes but very inaccurate in others, that model can still be used in those regimes where it is known to be very accurate.</p>

<p>A good example is Newtonian mechanics. Newtonian mechanics now known to be wrong (sometimes very wrong) as a universal model. Using Newtonian principles to build a collider that studies quantum events that result from relativistic collisions doesn't make much sense. On the other hand, Newtonian mechanics works very well in the ordinary, everyday world of earthly events. We still build bridges using Newtonian mechanics. It would be downright silly to invoke general relativity in bridge design.</p>

<p>What if the model doesn't work anywhere, or if the interval over which it does yield somewhat reasonable results is vanishingly small? In that case, the model isn't very good. A model needs to have some space over which the results are acceptable to have even limited applicability.</p>

CC BY-SA 3.0
2015-01-06T18:49:24.013Z
2015-01-06T18:49:24.013Z
David Hammen
<p>Say we have a model that describes a physical or dynamical system, for example, a hydrological model. If the model results do not match the measurements of the quantity being simulated, i.e. the model validation fails, is there any value in using the model for some other case or application?  </p>

CC BY-SA 3.0
2015-01-05T13:29:48.803Z
I don't think your understanding of the term "calibration" matches my own. In modelling (as opposed to the rest of science), it typically refers to tweaking input parameters to a model - usually ones that cannot be directly measured - until the model's predictions match observations. As such, I'm not sure how a model would "fail calibration", except perhaps if it was impossible to get a good match.
Semidiurnal Simon
CC BY-SA 3.0
2015-01-05T17:14:29.197Z
It seems as though he is asking: If results do not pass some sort of significance test is there any reason for analysis ?
Neo
CC BY-SA 3.0
2015-01-05T23:01:26.817Z
I edited the question based on my guess what the intent of the OP was. Please edit with more details if there are any. Voting to reopen.
milancurcic
CC BY-SA 3.0
2015-01-06T17:55:00.697Z
The original question has been lost....In modeling a calibration procedure must have a pass-fail criteria much like on a test.   If a certain level of reliability is not met the calibration of the model fails.  Using the test score as the example if one student gets 40% and another is 30% both are producing results but both are failing due to the passing grade being at least 50%.  My original question is if the model does not meet the reliability criteria is there any value in going from the calibration to the validation of the model?
user6784
CC BY-SA 3.0
2015-01-07T08:54:18.867Z
@user6784 in e.g. computer modelling reliability checks are there for a reason. So unless you really know what you are doing it doesn't seem wise to proceed further.
taupunkt
CC BY-SA 3.0
2015-01-07T13:50:51.237Z
@user6784 you say "In modeling a calibration procedure must have a pass-fail criteria". IME that's not the case at all. Having been involved in dozens of calibrations, including working with some of the world's experts in the field, I've never once seen an instance where the outcome of a calibration was judged as a pass/fail binary.
410 gone
CC BY-SA 3.0
2015-01-07T17:51:55.283Z
@EngergyNumber I can see how the wording I used was misleading.  Having a reliability criteria is used frequently as part of the methodology of evaluating model performance.  Like I indicated when interpreting results an acceptability criteria is important.
user6784
CC BY-SA 3.0
2015-01-07T22:52:38.837Z
@user6784 what you are describing as calibration seems like something that I would think of as validation. Perhaps you could clarify what you see the difference between the two as being? With that said, user6784 is clearly not happy with the edit that was made to his question, and possibly it shouldn't have been made as it made an assumption as to their intent.
Semidiurnal Simon
CC BY-SA 3.0
2015-01-08T06:27:42.230Z
However, as it stands, this is a *really good* question. It'd be a shame to lose it. @user6784: I'd suggest you leave this question alone, and ask a new, improved question, and reference this one in it.
naught101
CC BY-SA 3.0
2015-01-08T15:52:05.637Z
Yes I agree the question has generated a lot of response.  I can re-post an improved question.
user6784
CC BY-SA 3.0
2016-05-06T15:22:11.843Z
Don't all models fail (to some degree) when validating against measurements? Hence the tautology of wrong but useful models.
Antonio
<models>
Is there value in using a model that fails at validation against measurements?
user6784
# Is there value in using a model that fails at validation against measurements?

(Asked by: username_0 on 2015-01-04)

Say we have a model that describes a physical or dynamical system, for example, a hydrological model. If the model results do not match the measurements of the quantity being simulated, i.e. the model validation fails, is there any value in using the model for some other case or application?

**Tags:** models

> **Comments:**
> **username_1**: I don't think your understanding of the term "calibration" matches my own. In modelling (as opposed to the rest of science), it typically refers to tweaking input parameters to a model - usually ones that cannot be directly measured - until the model's predictions match observations. As such, I'm not sure how a model would "fail calibration", except perhaps if it was impossible to get a good match.  
> **username_2**: It seems as though he is asking: If results do not pass some sort of significance test is there any reason for analysis ?  
> **username_3**: I edited the question based on my guess what the intent of the OP was. Please edit with more details if there are any. Voting to reopen.  
> **username_0**: The original question has been lost....In modeling a calibration procedure must have a pass-fail criteria much like on a test.   If a certain level of reliability is not met the calibration of the model fails.  Using the test score as the example if one student gets 40% and another is 30% both are producing results but both are failing due to the passing grade being at least 50%.  My original question is if the model does not meet the reliability criteria is there any value in going from the calibration to the validation of the model?  
> **username_4**: @username_0 in e.g. computer modelling reliability checks are there for a reason. So unless you really know what you are doing it doesn't seem wise to proceed further.  
> **username_5**: @username_0 you say "In modeling a calibration procedure must have a pass-fail criteria". IME that's not the case at all. Having been involved in dozens of calibrations, including working with some of the world's experts in the field, I've never once seen an instance where the outcome of a calibration was judged as a pass/fail binary.  
> **username_0**: @EngergyNumber I can see how the wording I used was misleading.  Having a reliability criteria is used frequently as part of the methodology of evaluating model performance.  Like I indicated when interpreting results an acceptability criteria is important.  
> **username_1**: @username_0 what you are describing as calibration seems like something that I would think of as validation. Perhaps you could clarify what you see the difference between the two as being? With that said, username_0 is clearly not happy with the edit that was made to his question, and possibly it shouldn't have been made as it made an assumption as to their intent.  
> **username_6**: However, as it stands, this is a *really good* question. It'd be a shame to lose it. @username_0: I'd suggest you leave this question alone, and ask a new, improved question, and reference this one in it.  
> **username_0**: Yes I agree the question has generated a lot of response.  I can re-post an improved question.  
> **username_7**: Don't all models fail (to some degree) when validating against measurements? Hence the tautology of wrong but useful models.  

---

## 6 answers

### Answer by username_9 on 2015-01-06. Score: 9 | Accepted answer

The answer depends very much on where the model fails. If it is very accurate in some regimes but very inaccurate in others, that model can still be used in those regimes where it is known to be very accurate.

A good example is Newtonian mechanics. Newtonian mechanics now known to be wrong (sometimes very wrong) as a universal model. Using Newtonian principles to build a collider that studies quantum events that result from relativistic collisions doesn't make much sense. On the other hand, Newtonian mechanics works very well in the ordinary, everyday world of earthly events. We still build bridges using Newtonian mechanics. It would be downright silly to invoke general relativity in bridge design.

What if the model doesn't work anywhere, or if the interval over which it does yield somewhat reasonable results is vanishingly small? In that case, the model isn't very good. A model needs to have some space over which the results are acceptable to have even limited applicability.


---

### Answer by username_5 on 2015-01-06. Score: 11

The late George Box famously said:

> All models are wrong. Some models are useful

All models are wrong.

That's important.

*All* models are wrong.

In any real-world system that we're interested in, the only accurate model of the system, is the system itself. Anything else, any simplification, gives wrong answers.

Some models are useful.

That's important, too.

Some models help us structure our discussions of the uncertainties (H/T Prof Neil Strachan)

Some models allow us to test "what if" scenarios, and look at what the relative changes might be. Now, in the real world, exogenous influences that the model can't account for, would mean that the actual outcomes would be different: but if the model gives a near-enough account of the scale of relative changes, we can still make informed judgements on the basis of the models. Even when it involves pesky things such as turbulence.

Some models give us insights into emergent properties of systems: a lot of the really interesting problems are emergent properties that aren't necessarily obvious from just looking at the basic rules of the system, and any insights we can gain are valuable.

Some models allow us to do experiments on systems that we couldn't possibly experiment on. And although we know the models are wrong, they can at least give us an indication of the range of possible outcomes.

That's not to defend the use of bad models to do bad work. There's far too much of that in my field, and I do not like it. As username_1 W says, it is possible to test models to see if they are suitable for a particular purpose. To continue using a model *for a purpose where it is known to be inapplicable*, is charlatanry, not academia.

> **Comments:**
> **username_6**: All models are wrong, except, of course, [the big one](http://hitchhikers.wikia.com/wiki/Earth).  

---

### Answer by username_8 on 2015-01-05. Score: 8

Yes there is value.

1)Often times we use models to check our own understanding of a process. If the model is built with all known factors included, it might still fail to produce results that can be verified with measurements. When the model is unable to reproduce a phenomenon, the test case can be used to assess what parts of the model need further development.

2)Sometimes systems are too complicated to model accurately (or perhaps there are too many unknowns in the initial or boundary conditions). Although the model might fail to reproduce absolute values that are measured, it might still be able to produce realistic dynamic variance. So, the model is still useful for studying the relationships between the dynamic processes that are involved. This is sometimes how we find secondary effects that are not typically direct outcomes of the model processes.


---

### Answer by username_6 on 2015-01-08. Score: 7

Great question, with some interesting answers. It's obviously struck a chord around here.

I'd just like to add the abstract from one of my favourite papers, by Oreskes, Shrader-Frechette, & Beiitz (1994):

> Verification and validation of numerical models of natural systems is impossible. This is because natural systems are never closed and because model results are always non-unique. Models can be confirmed by the demonstration of agreement between observation and prediction, but confirmation is inherently partial. Complete confirmation is logically precluded by the fallacy of affirming the consequent and by incomplete access to natural phenomena. Models can only be evaluated in relative terms, and their predictive value is always open to question. The primary value of models is heuristic.

Basically, in a complex system, you'd never expect a model's results to match observed data - there is noise everywhere, but also these systems contain deterministic chaos. You can simulate the behaviour of such a system, but you can't simulate the *results*. Even if your model is exactly correct, with the correct inputs, you won't get the same behaviour, because even the slightest value truncation will lead to a significant divergence, eventually.

Also, even if you do get the *exact result* you can't know that you got the right result for the right reasons - equifinality is a real problem. *Especially* in a model with more than two interacting calibrated parameters. There is no guarantee that the calibration process will give you better parameters, you can only be sure that it will give you better results *for the calibration data*. Even calibration using a cross-validation methodology (train-and-test) with actually independent data (rare) might fail, because the validation part can fail for the above reasons.

Anyway, everyone interested in that this topic should read that paper, I'm sure I've missed points here. It is a bit heavy-going, but that's because of the concepts - the paper is actually pretty enjoyable to read.

### References

* Oreskes, N., Shrader-Frechette, K. & Beiitz, K., 1994. Verification, validation, and confirmation of numerical models in the Earth sciences. Science, 263(5147), pp.641–646. Available at: [ProQuest](http://search.proquest.com/docview/733194143/14125288294380DFD60/1?accountid=12763).


---

### Answer by username_1 on 2015-01-06. Score: 7

A slightly different slant:

As energynumbers has pointed out, all models are wrong in one way or another. A fully accurate model of reality would be as complex as reality, so some simplification is always needed. This means that there cannot be any useful model whose predictions in every way are perfectly accurate (except possibly by accident).

Validating a model, therefore, is not a process for saying "Is this model correct" with a yes/no answer. Rather, it should be about,

1. Deciding criteria the model must fulfill to be useful *for a given purpose*.
2. Assessing whether it fulfills those criteria.

**If the model fails validation for one application, it may still be useful for another.**

Some examples of different validation criteria, taking a regional ocean model as an example:

One obvious choice is whether the predictions of current speeds or of water levels is most important. Even after selecting one of these, the way in which the predictions are assessed may vary. A reasonable default might be to assess the fit at every time point between measured and observed water levels in a number of places, and accept the model if measurements such as bias and RMSE are below defined thresholds. However, here are two examples of specific scenarios that justify different approaches:

* NOAA runs models whose primary application is the production of depth information for navigational aids. The critical statistic for them is not the general accuracy of their models, but the frequency with which the model overpredicts the water level - as this form of error could result in vessels running aground. [1]
* Vested et al (1995) [2] give an example of a storm surge model that would provide flood warnings. It was tested not for the accuracy of all of its water level predictions, but for the accuracy of its predictions of *peak* water levels, as those are what would matter in operation.

[1] NOAA [NOS STANDARDS FOR EVALUATING OPERATIONAL NOWCAST AND FORECAST HYDRODYNAMIC MODEL SYSTEMS Silver Spring, Maryland October 2003 noaa National Oceanic and Atmospheric Administration U.S.](http://www.nauticalcharts.noaa.gov/csdl/docs/RD_standards_Hess_etal.pdf), NOAA Technical Report NOS CS 17, Oct. 2003

[2] H. J. Vested, J. W. Nielsen, H. R. Jensen, and K. B. Kristensen, “Skill Assessment of an Operational Hydrodynamic Forecast System for the North Sea and Danish Belts,” in Quantitative Skill Assessment for Coastal Ocean Models, vol. 47, D. R. Lynch and A. M. Davies, Eds. Washington DC: American Geophysical Union, 1995, pp. 373–396.


---

### Answer by username_4 on 2015-01-06. Score: 6

Since there are already two answers discussing possible merits in further analysis let me just add the other side of the coin; it could also be prudent to see the simulation as a failure and not analyse it any further.

That could be the case if it's state of the art that similar simulations reproduce these experimental values you were mentioning and if these simulation can reproduce the values you want to look at as well.
In that case, it only makes sense to try to find out why you can't reproduce your experimental test values. If you do not want to spend much time in developing the model, then drop it altogether; no one will trust your results.

Trust is, in my opinion, the crucial point here. As mentioned in the previous answers it is not uncommon for simulations to give only (poor) estimates of experimental results. But it is all important in these cases that you can scrutinize why you still trust the results you want to get out of it.

To be honest, I do read your question that way that you have no good reason to be confident in further results. If that is the case my advice would be to step back and first think about whether you can trust them. To do that no general answer will help you; it depends too strongly on your system and what the state of the art is for these kind of simulations.


---

