<p>Generally meteorologists use mean bias and mean error metrics to characterize the magnitude of their "success" for things like precipitation and wind speed.  Usually a monthly average of performance metrics (or even seasonal) would be used to summarize those results.  Also, it's important to keep in mind that for something like rain, they would be more concerned with the total per day (not the exact time that precip occurred).  Furthermore, keep in mind most weather performance is assessed at airports, where there are weather towers, so they don't assess their predictions over some large area, because they need instrument observations to do so.</p>

CC BY-SA 3.0
2014-05-30T23:16:20.833Z
2014-05-30T23:16:20.833Z
f.thorpe
<p>Disclaimer: I am not a meteorologist.</p>

<p>To answer this question, you need to understand how a forecast is obtained. Basically, meteorologists run computer simulations that predict how weather systems might evolve from current measurements. In fact, data assimilation techniques (<a href="http://en.wikipedia.org/wiki/Data_assimilation">http://en.wikipedia.org/wiki/Data_assimilation</a>) are commonly used, so that the predictions given by the model depend not only on current observations, but also upon the history of the weather system. As new data comes in, the evolving model is adjusted towards those values - but not forced to match exactly, as that would destroy the historical information.</p>

<p>Any one model is therefore not expected to be entirely accurate. However, if we run many models simultaneously, and include random variations to reflect the chaotic nature of the weather system (and measurement errors), we obtain something called an "ensemble forecast". This is where the 30% chance of rain comes from: 30% of the simulations predict rain in your area, while the others think it will remain dry. Average/minimum/maximum temperatures etc can also easily be extracted from the ensemble.</p>

<p>So, the accuracy of a forecast can be considered in different ways. It is certainly possible to compare the predictions of one ensemble member to observations: this is an integral part of the data assimilation process. However, it is not necessarily a problem if a member performs poorly on one particular occasion. Different types of measurements may be weighted differently during the comparison (so matching rainfall might be deemed 'more important' than matching temperature; this may vary depending on the type of forecast/its intended use). More generally, you have to look at the long-term statistics of the errors in the forecast. So, if you look at several months of data, did it rain on around a third of the days when you predicted a 30% chance of rain? On average, are your temperature predictions reasonably accurate? Making this assessment on a day-by-day basis is unlikely to be in keeping with the statistical basis of modern weather forecasting.</p>

CC BY-SA 3.0
2014-05-28T22:50:27.623Z
Good information, and obviously being "wrong" on a single day wouldn't be the end of a career (especially in [New England](https://en.wikipedia.org/wiki/April_Fool's_Day_Blizzard)). But, if being right "on the average" is the goal, then why not always predict the historical average conditions for that time of year? Is a forecast that is significantly different from the historical average given more credit when it turns out to be close to reality?
Daniel Griscom
CC BY-SA 3.0
2014-05-29T01:09:14.173Z
I'll add that there is more to producing a forecast than just regurgitating model output.  Model output and observations are a starting point, added with knowledge of the local weather patterns and the forecasters own interpretation of the dynamics.
casey
CC BY-SA 3.0
2014-05-29T11:40:10.747Z
@daniel I'm not sure I can say anything specific to the weather forecasting question, so I won't edit my answer. But generally with statistical methods, one looks at the difference between the prior probability distribution (what we knew before starting our analysis, so here that might be the range of historic measurements for a given location in a particular season) and the posterior probability distribution (obtained from our analysis). The posterior ought to be much narrower than the prior. This can be quantified by various measures of "information gain".
avid
CC BY-SA 3.0
2014-05-28T19:20:10.073Z
2014-05-28T19:44:41.087Z
2014-05-28T19:44:41.087Z
avid
<p>In every professional field, practitioners may come home at the end of the day and say "Yeah! I knocked it out of the park!", or alternatively "Oh, man. Gotta do better tomorrow." What about meteorologists?</p>

<p>When a meteorologist makes a weather forecast, she will strive to "get it right", so that when the date in question arrives her prediction will have matched the actual temperature, dew point, wind velocity and direction, precipitation, etc. But, when that date arrives, and the actual conditions have been measured and compared to the prediction, how is the prediction graded?</p>

<p>Precipitation seems to be the most problematic aspect, as it's quite nonlinear and predicted statistically. What does it mean to have "gotten it right" by predicting a 30% chance of precipitation? Yes, it rained 30% of the time? Yes, it rained in 30% of the forecast area? Yes, 30% of the public will answer "Yes" if asked whether it rained?</p>

<p>There's complexity in the other metrics as well. For instance, how are errors in the different values compared? Would it be better to get the temperature spot on but the wind velocity 5MPH high, or the wind exactly right but the temperature 5&deg;F high? What about timing: what if that front came through two hours after it was predicted?</p>

<p>Edit: part of the difficulty is getting clarity about the consumer of the weather information. Fred the Farmer wants to know how much rain he can expect in next day or so, but the exact timing probably isn't important. Pete the Party Planner wants to herd his guests into the bar before the front comes through, but if it's going to rain he probably doesn't care whether it drops a quarter inch or a half. And Francine the Fisherman doesn't really care about the precipitation at all (within limits); her world centers around wind and waves.</p>

<p>I may want to post a second question that is explicitly from the consumer viewpoint: given weather data and meteorologists' historical forecasts, how would I choose a meteorologist?</p>

CC BY-SA 3.0
2014-05-29T08:12:53.913Z
See also http://earthscience.stackexchange.com/q/295/124
BHF
CC BY-SA 3.0
2017-05-26T06:17:50.063Z
See https://www.weather.gov/mdl/verification_home for a great deal of verification statistics and insight.
JeopardyTempest
CC BY-SA 3.0
2017-05-26T12:39:54.373Z
@JeopardyTempest That's a great resource! You should consider making it into an answer.
Daniel Griscom
CC BY-SA 3.0
2017-05-26T12:42:54.380Z
@DanielGriscom If I can remember later on, I'll take a look at it.  The answers here honestly don't seem too precise and sharp, and though I don't regularly involve in verification, I know enough about it and should be able to dig up some details to give a hopefully more useful answer.
JeopardyTempest
CC BY-SA 3.0
2018-04-27T11:58:28.643Z
Here's a relevant resource: https://www.xkcd.com/1985/
Daniel Griscom
<weather><weather-forecasting>
How do meteorologists grade their past forecasts?
Daniel Griscom
# How do meteorologists grade their past forecasts?

(Asked by: username_0 on 2014-05-28)

In every professional field, practitioners may come home at the end of the day and say "Yeah! I knocked it out of the park!", or alternatively "Oh, man. Gotta do better tomorrow." What about meteorologists?

When a meteorologist makes a weather forecast, she will strive to "get it right", so that when the date in question arrives her prediction will have matched the actual temperature, dew point, wind velocity and direction, precipitation, etc. But, when that date arrives, and the actual conditions have been measured and compared to the prediction, how is the prediction graded?

Precipitation seems to be the most problematic aspect, as it's quite nonlinear and predicted statistically. What does it mean to have "gotten it right" by predicting a 30% chance of precipitation? Yes, it rained 30% of the time? Yes, it rained in 30% of the forecast area? Yes, 30% of the public will answer "Yes" if asked whether it rained?

There's complexity in the other metrics as well. For instance, how are errors in the different values compared? Would it be better to get the temperature spot on but the wind velocity 5MPH high, or the wind exactly right but the temperature 5°F high? What about timing: what if that front came through two hours after it was predicted?

Edit: part of the difficulty is getting clarity about the consumer of the weather information. Fred the Farmer wants to know how much rain he can expect in next day or so, but the exact timing probably isn't important. Pete the Party Planner wants to herd his guests into the bar before the front comes through, but if it's going to rain he probably doesn't care whether it drops a quarter inch or a half. And Francine the Fisherman doesn't really care about the precipitation at all (within limits); her world centers around wind and waves.

I may want to post a second question that is explicitly from the consumer viewpoint: given weather data and meteorologists' historical forecasts, how would I choose a meteorologist?

**Tags:** weather, weather-forecasting

> **Comments:**
> **username_1**: See also http://earthscience.stackexchange.com/q/295/124  
> **username_2**: See https://www.weather.gov/mdl/verification_home for a great deal of verification statistics and insight.  
> **username_0**: @username_2 That's a great resource! You should consider making it into an answer.  
> **username_2**: @username_0 If I can remember later on, I'll take a look at it.  The answers here honestly don't seem too precise and sharp, and though I don't regularly involve in verification, I know enough about it and should be able to dig up some details to give a hopefully more useful answer.  
> **username_0**: Here's a relevant resource: https://www.xkcd.com/1985/  

---

## 2 answers

### Answer by username_4 on 2014-05-28. Score: 8 | Accepted answer

Disclaimer: I am not a meteorologist.

To answer this question, you need to understand how a forecast is obtained. Basically, meteorologists run computer simulations that predict how weather systems might evolve from current measurements. In fact, data assimilation techniques (<http://en.wikipedia.org/wiki/Data_assimilation>) are commonly used, so that the predictions given by the model depend not only on current observations, but also upon the history of the weather system. As new data comes in, the evolving model is adjusted towards those values - but not forced to match exactly, as that would destroy the historical information.

Any one model is therefore not expected to be entirely accurate. However, if we run many models simultaneously, and include random variations to reflect the chaotic nature of the weather system (and measurement errors), we obtain something called an "ensemble forecast". This is where the 30% chance of rain comes from: 30% of the simulations predict rain in your area, while the others think it will remain dry. Average/minimum/maximum temperatures etc can also easily be extracted from the ensemble.

So, the accuracy of a forecast can be considered in different ways. It is certainly possible to compare the predictions of one ensemble member to observations: this is an integral part of the data assimilation process. However, it is not necessarily a problem if a member performs poorly on one particular occasion. Different types of measurements may be weighted differently during the comparison (so matching rainfall might be deemed 'more important' than matching temperature; this may vary depending on the type of forecast/its intended use). More generally, you have to look at the long-term statistics of the errors in the forecast. So, if you look at several months of data, did it rain on around a third of the days when you predicted a 30% chance of rain? On average, are your temperature predictions reasonably accurate? Making this assessment on a day-by-day basis is unlikely to be in keeping with the statistical basis of modern weather forecasting.

> **Comments:**
> **username_0**: Good information, and obviously being "wrong" on a single day wouldn't be the end of a career (especially in [New England](https://en.wikipedia.org/wiki/April_Fool's_Day_Blizzard)). But, if being right "on the average" is the goal, then why not always predict the historical average conditions for that time of year? Is a forecast that is significantly different from the historical average given more credit when it turns out to be close to reality?  
> **username_5**: I'll add that there is more to producing a forecast than just regurgitating model output.  Model output and observations are a starting point, added with knowledge of the local weather patterns and the forecasters own interpretation of the dynamics.  
> **username_4**: @username_0 I'm not sure I can say anything specific to the weather forecasting question, so I won't edit my answer. But generally with statistical methods, one looks at the difference between the prior probability distribution (what we knew before starting our analysis, so here that might be the range of historic measurements for a given location in a particular season) and the posterior probability distribution (obtained from our analysis). The posterior ought to be much narrower than the prior. This can be quantified by various measures of "information gain".  

---

### Answer by username_3 on 2014-05-30. Score: 6

Generally meteorologists use mean bias and mean error metrics to characterize the magnitude of their "success" for things like precipitation and wind speed. Usually a monthly average of performance metrics (or even seasonal) would be used to summarize those results. Also, it's important to keep in mind that for something like rain, they would be more concerned with the total per day (not the exact time that precip occurred). Furthermore, keep in mind most weather performance is assessed at airports, where there are weather towers, so they don't assess their predictions over some large area, because they need instrument observations to do so.


---

