<p>Climate projections shouldn't be seen as deterministic projection of climate. Since the earth system is non-linear (i.e. chaotic), a lot of path are possible for the next decades. In other words, the statistics of climate (i.e. trend, inter-annual variability, spatial patterns) of the next 3 decades could be closer to the 2060-2090 climate projections than the climate projection of 2020-2050. </p>

<p>This kind of comparison lies in the decadal prediction framework. This is an emerging science in itself. One that is far from being ready for prime time. The simulations done in the 1990s were not aimed at such timescale (decadal) prediction.</p>

<p>I doesn't answer directly the question, but I'm merely pointing out that such comparisons might be misleading.</p>

CC BY-SA 3.0
2015-09-21T10:16:06.200Z
While many ways of doing the comparisons will be misleading, surely we should expect some testing of the model's predictive power? Otherwise how do we know the models are not concealing some serious flaw?
matt_black
CC BY-SA 3.0
2015-09-22T11:00:37.777Z
@matt_black, I think Balinus is pointing out that a projection that is accurate on a centennial scale may not be very accurate on a decadal scale, as sources of internal climate variability (e.g. ENSO) still have a large effect at a decadal scale relative to the expected warming, but this averages out to a much greater extent on a centennial scale.  The point is, "how accurate can we reasonably expect the models to be on a decadal scale?".  A hint is given by the spread of the individual model runs forming the ensemble.
Dikran Marsupial
CC BY-SA 3.0
2014-08-08T17:42:49.857Z
2014-08-12T14:30:36.127Z
2014-08-12T14:30:36.127Z
Balinus
<p><strong>Some comparisons do suggest that historic models do a poor job of simulating observed temperatures</strong></p>

<p>This picture from a <a href="http://www.nature.com/nclimate/journal/v3/n9/full/nclimate1972.html?WT.ec_id=NCLIMATE-201309" rel="nofollow noreferrer">Nature article</a> is the clearest analysis that avoids the various obfuscations possible when looking at noisy time series data:</p>

<p><a href="https://i.sstatic.net/VYfUt.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/VYfUt.jpg" alt="nature chart"></a></p>

<p>The choice made by these authors is to compare just the average extent of warming over a period rather than the noisy time series. This appears to avoid some of the choices that make the time series comparison so obfuscating and controversial (in fact their results don't disagree much with the ones shown in the previous answer form the IPCC, they just choose a presentation method that emphasises the comparison between models and observations). Their result seems to show that most models really do overestimate the actual amount of warming over recent years. In their words:</p>

<blockquote>
  <p>Recent observed global warming is significantly less than that simulated by climate models.</p>
</blockquote>

<p>In somewhat more detail the authors talk about the statistics like this (my highlights):</p>

<blockquote>
  <p>The evidence, therefore, <em>indicates that the current generation of climate models (when run as a group, with the CMIP5 prescribed forcings) do not reproduce the observed global warming over the past 20 years, or the slowdown in global warming over the past fifteen years</em>. This interpretation is supported by statistical tests of the null hypothesis that the observed and model mean trends are equal, assuming that either: (1) the models are exchangeable with each other (that is, the ‘truth plus error’ view); or (2) the models are exchangeable with each other and with the observations (see Supplementary Information). Differences between observed and simulated 20-year trends have p values (Supplementary Information) that drop to close to zero by 1993–2012 under assumption (1) and to 0.04 under assumption (2) (Fig. 2c). Here we note that the smaller the p value is, the stronger the evidence against the null hypothesis. On this basis, the rarity of the 1993–2012 trend difference under assumption (1) is obvious. Under assumption (2), this implies that <em>such an inconsistency is only expected to occur by chance once in 500 years</em>, if 20-year periods are considered statistically independent. Similar results apply to trends for 1998–2012. In conclusion, we reject the null hypothesis that the observed and model mean trends are equal.</p>
</blockquote>

<p>The conclusion is that most models (or, perhaps, most results from the ensemble of models) predict more warming that has actually happened. Whether this is a significant problems for models will become more obvious in the next few years as observational data accumulates. The authors include some caveats:</p>

<blockquote>
  <p>Other factors that contribute to the discrepancy could include a missing decrease in stratospheric water vapour (whose processes are
  not well represented in current climate models), errors in aerosol forcing in the CMIP5 models, a bias in the prescribed solar irradiance trend, the possibility that the transient climate sensitivity of the CMIP5 models could be on average too high or a possible unusual episode of internal climate variability not considered above. Ultimately the causes of this inconsistency will only be understood after careful comparison of simulated internal climate variability and climate model forcings with observations from the past two decades, and by waiting to see how global temperature responds over the coming decades.</p>
</blockquote>

<p>Other authors have addressed the question using a slightly different and less direct approach. They tend to compare the implied climate sensitivity (crudely: how much warming we expect from a doubling of CO<sub>2</sub> levels in the short or long term) from observations and models. Several recent papers have shown lower estimates (especially for transient climate response) than the values in climate models (see <a href="http://link.springer.com/article/10.1007%2Fs00382-014-2342-y#page-1" rel="nofollow noreferrer">Lewis &amp; Curry</a>; <a href="https://www.researchgate.net/publication/260740743_Bayesian_estimation_of_climate_sensitivity_based_on_a_simple_climate_model_fitted_to_observations_of_hemispheric_temperatures_and_global_ocean_heat_content" rel="nofollow noreferrer">Aldrin et. al.</a>; <a href="http://www.iac.ethz.ch/people/knuttir/papers/otto13nat.pdf" rel="nofollow noreferrer">Otto et. al.</a>; <a href="http://www.earth-syst-dynam.net/5/139/2014/esd-5-139-2014.html" rel="nofollow noreferrer">Skeie et. al.</a>.</p>

<p><strong>Summary</strong>
The question was whether models do a good job of simulating the actual future climate. These results suggest they haven't. But there is plenty of scientific uncertainty about whether that failure is <em>significant</em>. </p>

CC BY-SA 3.0
2015-09-22T11:06:37.717Z
A key problem with this paper is that it uses the model variability to estimate the expected variability of the observations (effectively), which means you can't claim that the models are running too warm, only that they are either running too warm, or they under-simulate natural variability or a bit of both.  My opinion it is a bit of both.  The paper isn't that controversial, the problem is the overstating of the significance on climate blogs (e.g. the first line of your answer, which is written in bold).
Dikran Marsupial
CC BY-SA 3.0
2015-09-22T11:12:30.110Z
" In conclusion, we reject the null hypothesis that the observed and model mean trends are equal." which we wouldn't expect anyway.  The ensemble mean is an estimate of the forced response of the climate system, whereas the observed trend is the result of the forced response AND a realization of the unforced response (i.e. weather noise), and so the two will only be the same if the weather noise is very small.  Proper test is to see if the observed trend falls within the spread of the modeled trends, which it apparently does.
Dikran Marsupial
CC BY-SA 3.0
2015-09-22T19:39:26.407Z
@DikranMarsupial The question they were trying to answer was whether it was reasonable that the noise in the system would lead to actual weather being a long way from forecast weather over an extended period. They concluded that this was unlikely. You disagree. So what test would you propose for the validity of models?
matt_black
CC BY-SA 3.0
2015-09-24T07:34:47.400Z
of course it is a reasonable question to try to answer, there has been a stack of research on this.  We are also talking about climate here, not weather.  The appropriate test of model-observation consistency (model validity cannot be answered by a single statistical test - see the relevant chapters in IPCC reports) is to determine whether the observations lie within the spread of the model runs, following appropriate baselining.  Note that nobody expects the individual models to be correct, the multi-model ensemble tries to represent uncertainty on the physics.
Dikran Marsupial
CC BY-SA 3.0
2015-09-24T07:36:51.623Z
"it was reasonable that the noise in the system would lead to actual weather being a long way from forecast weather over an extended period. " climate projections are not based on forecasting weather, but simulating weather.  There is a big difference between the two.
Dikran Marsupial
CC BY-SA 3.0
2015-09-24T08:19:40.947Z
Note however, the spread of the ensemble is likely to under-represent the true uncertainty, see this rather timely blog post by http://variable-variability.blogspot.co.uk/2015/09/model-spread-is-not-uncertainty-nwp.html (n.b. the author is a climatologist).
Dikran Marsupial
CC BY-SA 3.0
2015-09-25T09:52:41.943Z
@DikranMarsupial My careless use of language (simulation vs forecast and weather vs climate) does not detract form the central point: the only way we can know models are any good is to compare them to reality which is what the paper does.
matt_black
CC BY-SA 3.0
2015-09-25T12:16:56.347Z
there are very big differences between those two pairs, it would be a good idea to make sure you understand the difference if you are interested in comparing models and observations.  Having said which, it is notable that your answer doesn't include the many caveats that were included in the Fyfe paper and overstate its conclusions.  Note the paper ends "Ultimately the causes of this inconsistency will only be understood after careful comparison of simulated internal climate variability and climate model forcings with observations from the past two decades, ...
Dikran Marsupial
CC BY-SA 3.0
2015-09-25T12:18:09.507Z
... and by waiting to see how global temperature responds over the coming decades.".  In other words, they point out that it is not necessarily the models that are wrong, but that the modelled variability might be wrong or the problems with the model forcings.  If you were to re-write the answer, to include the caveats, put it into the context of the other studies on this and not overstate their conclusions, I'd happily upvote this answer (and not have the "dismissed out of hand" rhetoric at the end, it doesn't help make your point).
Dikran Marsupial
CC BY-SA 3.0
2015-09-21T10:13:31.093Z
2015-10-05T17:28:41.830Z
2015-10-05T17:28:41.830Z
matt_black
<p>You might take a look at <a href="http://www.climatechange2013.org/images/report/WG1AR5_TS_FINAL.pdf" rel="noreferrer">the Technical Summary of the most recent IPCC Assessment Report (5).</a></p>
<p>Thematic Focus Element #3 is &quot;Comparing Projections from Previous IPCC Assessments with Observations&quot;</p>
<p>It says:</p>
<blockquote>
<h3>Global Mean Temperature Anomaly</h3>
<p>Relative to the 1961–1990 mean, the GMST anomaly has been positive and
larger than 0.25°C since 2001. Observations are generally well within
the range of the extent of the earlier IPCC projections (TFE.3,
Figure1, middle left) This is also true for the Coupled Model
Intercomparison Project Phase 5 (CMIP5) results (TFE.3, Figure 1;
middle right) in the sense that the observed record lies within the
range of the model projections, but on the lower end of the plume. Mt
Pinatubo erupted in 1991 (see FAQ 11.2 for discussion of how volcanoes
impact the climate system), leading to a brief period of relative
global mean cooling during the early 1990s. The IPCC First, Second and
Third Assessment Reports (FAR, SAR and TAR) did not include the
effects of volcanic eruptions and thus failed to include the cooling
associated with the Pinatubo eruption. AR4 and AR5, however, did
include the effects from volcanoes and did simulate successfully the
associated cooling. During 1995–2000 the global mean temperature
anomaly was quite variable—a significant fraction of this variability
was due to the large El Niño in 1997–1998 and the strong back-to-back
La Niñas in 1999–2001. The projections associated with these
assessment reports do not attempt to capture the actual evolution of
these El Niño and La Niña events, but include them as a source of
uncertainty due to natural variability as encompassed by, for example,
the range given by the individual CMIP3 and CMIP5 simulations and
projection (TFE.3, Figure 1). The grey wedge in TFE.3, Figure 1
(middle right) corresponds to the indicative <em>likely</em> range for annual
temperatures, which is determined from the Representative
Concentration Pathways (RCPs) assessed value for the 20-year mean
2016–2035 (see discussion of Figure TS.14 and Section 11.3.6 for
details). From 1998 to 2012 the observational estimates have largely
been on the low end of the range given by the scenarios alone in
previous assessment reports and CMIP3 and CMIP5 projections. {2.4; Box
9.2}</p>
</blockquote>
<p>I've attached an image as well from that summary:</p>
<blockquote>
<p><img src="https://i.sstatic.net/rQPb0.png" alt="previous IPCC temperature performance" /></p>
<p>Estimated changes in the
observed globally and annually averaged surface temperature anomaly
relative to 1961–1990 (in °C) since 1950 compared with the range of
projections from the previous IPCC assessments. Values are harmonized
to start from the same value at 1990. Observed global annual
temperature anomaly, relative to 1961–1990, from three data sets is
shown as squares and smoothed time series as solid lines from the
Hadley Centre/Climatic Research Unit gridded surface temperature data
set 4 (HadCRUT4; bright green), Merged Land–Ocean Surface Temperature
Analysis (MLOST; warm mustard) and Goddard Institute for Space Studies
Surface Temperature Analysis (GISTEMP; dark blue) data sets. The
coloured shading shows the projected range of global annual mean near
surface temperature change from 1990 to 2035 for models used in FAR
(Figure 6.11), SAR (Figure 19 in the TS of IPCC 1996), TAR (full range
of TAR, Figure 9.13(b)). TAR results are based on the simple climate
model analyses presented in this assessment and not on the individual
full three-dimensional climate model simulations. For the AR4 results
are presented as single model runs of the CMIP3 ensemble for the
historical period from 1950 to 2000 (light grey lines) and for three
SRES scenarios (A2, A1B and B1) from 2001 to 2035. For the three SRES
scenarios the bars show the CMIP3 ensemble mean and the likely range
given by –40 % to +60% of the mean as assessed in Chapter 10 of AR4.</p>
</blockquote>

CC BY-SA 3.0
2015-10-09T21:46:38.070Z
Was that a yes or a no?
matt_black
CC BY-SA 3.0
2015-10-09T22:21:52.107Z
The long term temperature trends were within the range of the ensembles, as can be seen in the graph.
f.thorpe
CC BY-SA 4.0
2022-01-09T15:31:21.480Z
Is there a newer publication with a plot using observations until 2020?
asmaier
CC BY-SA 4.0
2022-01-09T19:09:56.473Z
It's still technically in draft form, and I don't see the same plot, but here is the AR6 Technical Summary:  https://www.ipcc.ch/report/ar6/wg1/downloads/report/IPCC_AR6_WGI_TS.pdf
f.thorpe
CC BY-SA 3.0
2014-06-25T20:01:36.367Z
2014-08-24T01:18:14.290Z
2020-06-18T08:25:19.090Z
f.thorpe
<p>Was there a significant bias between the ensemble of climate models and the long-term temperature trends?</p>

<p>What about spatial patterns?</p>

CC BY-SA 3.0
2015-09-24T16:50:17.907Z
Even the 2014 models, which are much better than the 1990s models, aren't very good at predicting 10-20 years in the future because the climate change in 10-20 years is only marginally bigger than the natural variation.   Statistical noise makes near term predictions difficult.  That said, my understanding of the 90s models is that they were too aggressive, not deliberately, but it was a new science.  They didn't know how good the oceans would be at absorbing heat trapped by Greenhouse gas.   I don't believe there was bias, though in some individuals maybe.  it was new and complicated science.
userLTK
CC BY-SA 3.0
2015-10-06T07:26:50.010Z
If interested, this article is related and goes into some detail.   http://www.skepticalscience.com/ipcc-overestimate-global-warming.htm
userLTK
CC BY-SA 3.0
2014-06-25T14:15:13.957Z
Climate is the statistics of the atmosphere over a period of ~30 years.  So it's sort of too early to tell.
gerrit
CC BY-SA 3.0
2014-06-26T12:01:20.177Z
In addition to @gerrit's observations, the models of the 1990s were not intended to be able to predict climate on a decadal scale, as this requires predicting sources of internal variability, such as ENSO.  As I understand it, the current generation of models is just beginning to reach the point where decadal projections are worth evaluating (CMIP5?).
Dikran Marsupial
<meteorology><atmosphere><climate><climate-change><climate-models>
How good were climate models of the 1990s at predicting the global temperature trends of the 2000s/2010s?
InquilineKea
# How good were climate models of the 1990s at predicting the global temperature trends of the 2000s/2010s?

(Asked by: username_0 on 2014-06-25)

Was there a significant bias between the ensemble of climate models and the long-term temperature trends?

What about spatial patterns?

**Tags:** meteorology, atmosphere, climate, climate-change, climate-models

> **Comments:**
> **username_2**: Climate is the statistics of the atmosphere over a period of ~30 years.  So it's sort of too early to tell.  
> **username_3**: In addition to @username_2's observations, the models of the 1990s were not intended to be able to predict climate on a decadal scale, as this requires predicting sources of internal variability, such as ENSO.  As I understand it, the current generation of models is just beginning to reach the point where decadal projections are worth evaluating (CMIP5?).  
> **username_1**: Even the 2014 models, which are much better than the 1990s models, aren't very good at predicting 10-20 years in the future because the climate change in 10-20 years is only marginally bigger than the natural variation.   Statistical noise makes near term predictions difficult.  That said, my understanding of the 90s models is that they were too aggressive, not deliberately, but it was a new science.  They didn't know how good the oceans would be at absorbing heat trapped by Greenhouse gas.   I don't believe there was bias, though in some individuals maybe.  it was new and complicated science.  
> **username_1**: If interested, this article is related and goes into some detail.   http://www.skepticalscience.com/ipcc-overestimate-global-warming.htm  

---

## 3 answers

### Answer by username_6 on 2014-06-25. Score: 17 | Accepted answer

You might take a look at [the Technical Summary of the most recent IPCC Assessment Report (5).](http://www.climatechange2013.org/images/report/WG1AR5_TS_FINAL.pdf)

Thematic Focus Element #3 is "Comparing Projections from Previous IPCC Assessments with Observations"

It says:

> ### Global Mean Temperature Anomaly
>
> Relative to the 1961–1990 mean, the GMST anomaly has been positive and
> larger than 0.25°C since 2001. Observations are generally well within
> the range of the extent of the earlier IPCC projections (TFE.3,
> Figure1, middle left) This is also true for the Coupled Model
> Intercomparison Project Phase 5 (CMIP5) results (TFE.3, Figure 1;
> middle right) in the sense that the observed record lies within the
> range of the model projections, but on the lower end of the plume. Mt
> Pinatubo erupted in 1991 (see FAQ 11.2 for discussion of how volcanoes
> impact the climate system), leading to a brief period of relative
> global mean cooling during the early 1990s. The IPCC First, Second and
> Third Assessment Reports (FAR, SAR and TAR) did not include the
> effects of volcanic eruptions and thus failed to include the cooling
> associated with the Pinatubo eruption. AR4 and AR5, however, did
> include the effects from volcanoes and did simulate successfully the
> associated cooling. During 1995–2000 the global mean temperature
> anomaly was quite variable—a significant fraction of this variability
> was due to the large El Niño in 1997–1998 and the strong back-to-back
> La Niñas in 1999–2001. The projections associated with these
> assessment reports do not attempt to capture the actual evolution of
> these El Niño and La Niña events, but include them as a source of
> uncertainty due to natural variability as encompassed by, for example,
> the range given by the individual CMIP3 and CMIP5 simulations and
> projection (TFE.3, Figure 1). The grey wedge in TFE.3, Figure 1
> (middle right) corresponds to the indicative *likely* range for annual
> temperatures, which is determined from the Representative
> Concentration Pathways (RCPs) assessed value for the 20-year mean
> 2016–2035 (see discussion of Figure TS.14 and Section 11.3.6 for
> details). From 1998 to 2012 the observational estimates have largely
> been on the low end of the range given by the scenarios alone in
> previous assessment reports and CMIP3 and CMIP5 projections. {2.4; Box
> 9.2}

I've attached an image as well from that summary:

> ![previous IPCC temperature performance](https://i.sstatic.net/rQPb0.png)
>
> Estimated changes in the
> observed globally and annually averaged surface temperature anomaly
> relative to 1961–1990 (in °C) since 1950 compared with the range of
> projections from the previous IPCC assessments. Values are harmonized
> to start from the same value at 1990. Observed global annual
> temperature anomaly, relative to 1961–1990, from three data sets is
> shown as squares and smoothed time series as solid lines from the
> Hadley Centre/Climatic Research Unit gridded surface temperature data
> set 4 (HadCRUT4; bright green), Merged Land–Ocean Surface Temperature
> Analysis (MLOST; warm mustard) and Goddard Institute for Space Studies
> Surface Temperature Analysis (GISTEMP; dark blue) data sets. The
> coloured shading shows the projected range of global annual mean near
> surface temperature change from 1990 to 2035 for models used in FAR
> (Figure 6.11), SAR (Figure 19 in the TS of IPCC 1996), TAR (full range
> of TAR, Figure 9.13(b)). TAR results are based on the simple climate
> model analyses presented in this assessment and not on the individual
> full three-dimensional climate model simulations. For the AR4 results
> are presented as single model runs of the CMIP3 ensemble for the
> historical period from 1950 to 2000 (light grey lines) and for three
> SRES scenarios (A2, A1B and B1) from 2001 to 2035. For the three SRES
> scenarios the bars show the CMIP3 ensemble mean and the likely range
> given by –40 % to +60% of the mean as assessed in Chapter 10 of AR4.

> **Comments:**
> **username_5**: Was that a yes or a no?  
> **username_6**: The long term temperature trends were within the range of the ensembles, as can be seen in the graph.  
> **username_7**: Is there a newer publication with a plot using observations until 2020?  
> **username_6**: It's still technically in draft form, and I don't see the same plot, but here is the AR6 Technical Summary:  https://www.ipcc.ch/report/ar6/wg1/downloads/report/IPCC_AR6_WGI_TS.pdf  

---

### Answer by username_4 on 2014-08-08. Score: 4

Climate projections shouldn't be seen as deterministic projection of climate. Since the earth system is non-linear (i.e. chaotic), a lot of path are possible for the next decades. In other words, the statistics of climate (i.e. trend, inter-annual variability, spatial patterns) of the next 3 decades could be closer to the 2060-2090 climate projections than the climate projection of 2020-2050.

This kind of comparison lies in the decadal prediction framework. This is an emerging science in itself. One that is far from being ready for prime time. The simulations done in the 1990s were not aimed at such timescale (decadal) prediction.

I doesn't answer directly the question, but I'm merely pointing out that such comparisons might be misleading.

> **Comments:**
> **username_5**: While many ways of doing the comparisons will be misleading, surely we should expect some testing of the model's predictive power? Otherwise how do we know the models are not concealing some serious flaw?  
> **username_3**: @username_5, I think username_4 is pointing out that a projection that is accurate on a centennial scale may not be very accurate on a decadal scale, as sources of internal climate variability (e.g. ENSO) still have a large effect at a decadal scale relative to the expected warming, but this averages out to a much greater extent on a centennial scale.  The point is, "how accurate can we reasonably expect the models to be on a decadal scale?".  A hint is given by the spread of the individual model runs forming the ensemble.  

---

### Answer by username_5 on 2015-09-21. Score: 0

**Some comparisons do suggest that historic models do a poor job of simulating observed temperatures**

This picture from a [Nature article](http://www.nature.com/nclimate/journal/v3/n9/full/nclimate1972.html?WT.ec_id=NCLIMATE-201309) is the clearest analysis that avoids the various obfuscations possible when looking at noisy time series data:

[![nature chart](https://i.sstatic.net/VYfUt.jpg)](https://i.sstatic.net/VYfUt.jpg)

The choice made by these authors is to compare just the average extent of warming over a period rather than the noisy time series. This appears to avoid some of the choices that make the time series comparison so obfuscating and controversial (in fact their results don't disagree much with the ones shown in the previous answer form the IPCC, they just choose a presentation method that emphasises the comparison between models and observations). Their result seems to show that most models really do overestimate the actual amount of warming over recent years. In their words:

> Recent observed global warming is significantly less than that simulated by climate models.

In somewhat more detail the authors talk about the statistics like this (my highlights):

> The evidence, therefore, *indicates that the current generation of climate models (when run as a group, with the CMIP5 prescribed forcings) do not reproduce the observed global warming over the past 20 years, or the slowdown in global warming over the past fifteen years*. This interpretation is supported by statistical tests of the null hypothesis that the observed and model mean trends are equal, assuming that either: (1) the models are exchangeable with each other (that is, the ‘truth plus error’ view); or (2) the models are exchangeable with each other and with the observations (see Supplementary Information). Differences between observed and simulated 20-year trends have p values (Supplementary Information) that drop to close to zero by 1993–2012 under assumption (1) and to 0.04 under assumption (2) (Fig. 2c). Here we note that the smaller the p value is, the stronger the evidence against the null hypothesis. On this basis, the rarity of the 1993–2012 trend difference under assumption (1) is obvious. Under assumption (2), this implies that *such an inconsistency is only expected to occur by chance once in 500 years*, if 20-year periods are considered statistically independent. Similar results apply to trends for 1998–2012. In conclusion, we reject the null hypothesis that the observed and model mean trends are equal.

The conclusion is that most models (or, perhaps, most results from the ensemble of models) predict more warming that has actually happened. Whether this is a significant problems for models will become more obvious in the next few years as observational data accumulates. The authors include some caveats:

> Other factors that contribute to the discrepancy could include a missing decrease in stratospheric water vapour (whose processes are
> not well represented in current climate models), errors in aerosol forcing in the CMIP5 models, a bias in the prescribed solar irradiance trend, the possibility that the transient climate sensitivity of the CMIP5 models could be on average too high or a possible unusual episode of internal climate variability not considered above. Ultimately the causes of this inconsistency will only be understood after careful comparison of simulated internal climate variability and climate model forcings with observations from the past two decades, and by waiting to see how global temperature responds over the coming decades.

Other authors have addressed the question using a slightly different and less direct approach. They tend to compare the implied climate sensitivity (crudely: how much warming we expect from a doubling of CO2 levels in the short or long term) from observations and models. Several recent papers have shown lower estimates (especially for transient climate response) than the values in climate models (see [Lewis & Curry](http://link.springer.com/article/10.1007%2Fs00382-014-2342-y#page-1); [Aldrin et. al.](https://www.researchgate.net/publication/260740743_Bayesian_estimation_of_climate_sensitivity_based_on_a_simple_climate_model_fitted_to_observations_of_hemispheric_temperatures_and_global_ocean_heat_content); [Otto et. al.](http://www.iac.ethz.ch/people/knuttir/papers/otto13nat.pdf); [Skeie et. al.](http://www.earth-syst-dynam.net/5/139/2014/esd-5-139-2014.html).

**Summary**
The question was whether models do a good job of simulating the actual future climate. These results suggest they haven't. But there is plenty of scientific uncertainty about whether that failure is *significant*.

> **Comments:**
> **username_3**: A key problem with this paper is that it uses the model variability to estimate the expected variability of the observations (effectively), which means you can't claim that the models are running too warm, only that they are either running too warm, or they under-simulate natural variability or a bit of both.  My opinion it is a bit of both.  The paper isn't that controversial, the problem is the overstating of the significance on climate blogs (e.g. the first line of your answer, which is written in bold).  
> **username_3**: " In conclusion, we reject the null hypothesis that the observed and model mean trends are equal." which we wouldn't expect anyway.  The ensemble mean is an estimate of the forced response of the climate system, whereas the observed trend is the result of the forced response AND a realization of the unforced response (i.e. weather noise), and so the two will only be the same if the weather noise is very small.  Proper test is to see if the observed trend falls within the spread of the modeled trends, which it apparently does.  
> **username_5**: @username_3 The question they were trying to answer was whether it was reasonable that the noise in the system would lead to actual weather being a long way from forecast weather over an extended period. They concluded that this was unlikely. You disagree. So what test would you propose for the validity of models?  
> **username_3**: of course it is a reasonable question to try to answer, there has been a stack of research on this.  We are also talking about climate here, not weather.  The appropriate test of model-observation consistency (model validity cannot be answered by a single statistical test - see the relevant chapters in IPCC reports) is to determine whether the observations lie within the spread of the model runs, following appropriate baselining.  Note that nobody expects the individual models to be correct, the multi-model ensemble tries to represent uncertainty on the physics.  
> **username_3**: "it was reasonable that the noise in the system would lead to actual weather being a long way from forecast weather over an extended period. " climate projections are not based on forecasting weather, but simulating weather.  There is a big difference between the two.  
> **username_3**: Note however, the spread of the ensemble is likely to under-represent the true uncertainty, see this rather timely blog post by http://variable-variability.blogspot.co.uk/2015/09/model-spread-is-not-uncertainty-nwp.html (n.b. the author is a climatologist).  
> **username_5**: @username_3 My careless use of language (simulation vs forecast and weather vs climate) does not detract form the central point: the only way we can know models are any good is to compare them to reality which is what the paper does.  
> **username_3**: there are very big differences between those two pairs, it would be a good idea to make sure you understand the difference if you are interested in comparing models and observations.  Having said which, it is notable that your answer doesn't include the many caveats that were included in the Fyfe paper and overstate its conclusions.  Note the paper ends "Ultimately the causes of this inconsistency will only be understood after careful comparison of simulated internal climate variability and climate model forcings with observations from the past two decades, ...  
> **username_3**: ... and by waiting to see how global temperature responds over the coming decades.".  In other words, they point out that it is not necessarily the models that are wrong, but that the modelled variability might be wrong or the problems with the model forcings.  If you were to re-write the answer, to include the caveats, put it into the context of the other studies on this and not overstate their conclusions, I'd happily upvote this answer (and not have the "dismissed out of hand" rhetoric at the end, it doesn't help make your point).  

---

