<p>I've been working with ICON, ICON-EU and GFS forecasts, and it seems like the temperature results are &quot;close&quot; for the same location, 6 hour forecasts and 120 hour forecasts - MAE: 5.7 and 6.87 respectively. Is it usually the case and if it is, I would like to find out the reasons briefly.</p>
<p>I'm fetching forecasts from global models every 6 hours. Forecasts are in the range of 0-144 hours, hourly incremented. I calculated the error metric by using only 6 hour forecasts vs 120 hour forecasts.</p>
<p>Dataset example below:
You can see that models forecast the next 144 hours in every run and I fetch every 6 hours so base_run_timestamp increments every 6 hours.
(Base_run_timestamp is when the models made their forecasts and forecasted_timestamp is pretty self-explanatory)</p>
<p>Note: I realized this is a consistent error for my observation point, I applied simple linear regression and the error went straight down to ~1°C so high error is irrelevant.</p>
<p>Thank you!</p>
<p><a href="https://i.sstatic.net/dkDEM.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/dkDEM.png" alt="DS_1" /></a> <a href="https://i.sstatic.net/I6RA3.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/I6RA3.png" alt="DS_2" /></a></p>

CC BY-SA 4.0
2022-03-22T13:37:16.720Z
I think I can answer your question if you provide a little more detail. What is the time step for the different model runs (is it even different?) ? What is the origin of your numbers? A mean average error of 5.7°C for a 6 hour forecast seems far from reasonable to me. Which time stepping method was deployed (this is the most important detail) ?
Joscha Fregin
CC BY-SA 4.0
2022-04-21T12:40:50.537Z
@J.Fregin Sorry for the late response and thanks for your interest. I've updated the question with examples.
UgurZCifci
CC BY-SA 4.0
2022-04-21T13:26:10.833Z
Sounds like you are probably in an area with significant terrain variation over short distance.  And so you're at a different elevation than your gridpoint?  Hopefully regression can work well often (though I'd expect there may be times it doesn't, times when the weather varies more by elevation)
JeopardyTempest
CC BY-SA 4.0
2022-04-24T12:30:19.283Z
I think otherwise the basic key questions are to try to make sure everything is being done properly... how big a dataset are you using and what data source are using to verify against the forecast.  Also, why are you doing just 6 hr and 120 hr, if you have a full database and are well familiar with this stuff, I would think you could pretty easily code to score all the forecasts and that might point you towards any issues you're having.
JeopardyTempest
CC BY-SA 4.0
2022-04-24T12:33:12.283Z
Because even for a place with a systemic error like poor terrain resolution, the degree of error should change in most situations over that range just because models are poorer at longer range.  And you're also comparing two different times of day (6 hours separate), so even if it were something straightforward like elevation/weather, that shouldn't be as constant for different times of day necessarily.
JeopardyTempest
CC BY-SA 4.0
2022-04-24T12:33:53.703Z
And perhaps explain this linear regression more.  How are you applying that?  To all data points in the forecast using all fhours?
JeopardyTempest
CC BY-SA 4.0
2022-04-25T05:46:20.253Z
@JeopardyTempest Thanks for your interest. Observation data comes from national meteorological service stations (sensor data) and the dataset we're using has a time range of 1 year. We've decided to measure our accuracy in different timesteps such as: 6-12-24-36-48-60... until 120 hours, could you elaborate on scoring all the forecasts a bit?
UgurZCifci
CC BY-SA 4.0
2022-04-25T07:25:33.300Z
ok, that makes sense (would've been a very odd error to have if you were comparing to F0).  And a fine sized sample (I had some concern you might be a student that was trying like one single model run, and of course that wouldn't be representative!).
JeopardyTempest
CC BY-SA 4.0
2022-04-25T07:26:54.483Z
So you're getting this kind of trend for multiple stations (as you said stations)?  That is peculiar.  Perhaps it would be good to compare to reality, we could like at current runs and see if that trend is continuing?  The more results you can offer, the better.
JeopardyTempest
CC BY-SA 4.0
2022-04-25T07:48:02.730Z
I'm not sure what you mean by elaborate on scoring the forecasts, indeed what you sound like you're trying to do makes fine sense.  We do the same thing in the US to a fair degree... though I can rarely dig up the sites... [here was a comparison of GFS MOS temperature errors](https://www.nws.noaa.gov/mdl/synop/gfspara07/warm/00mxmaeovrall.png).  Note that we do create MOS over here, which as far as I understand it tweaks the GFS basic data to account for station details and give a full weather forecast.
JeopardyTempest
CC BY-SA 4.0
2022-04-25T07:48:24.473Z
None of this is stuff I've messed with in a long while, but it is still surprising to have that consistent of an error over many sites.
JeopardyTempest
CC BY-SA 4.0
2022-04-26T12:29:20.447Z
Great, good to learn about MOS and comparison plot was helpful, thanks! We are going with a similar idea actually. Our forecast locations are settlement-based points rather than grids so we're tweaking the nearest grid point forecasts to our location. I'm working with solid meteorology engineers but this is my first time in this domain so I probably need a powerful visualization :) I'll dig deeper into our accuracy results, thanks!
UgurZCifci
<meteorology><temperature><weather-forecasting><gfs>
How much does a global model's accuracy change with the timestep?
UgurZCifci
# How much does a global model's accuracy change with the timestep?

(Asked by: username_0 on 2022-03-16)

I've been working with ICON, ICON-EU and GFS forecasts, and it seems like the temperature results are "close" for the same location, 6 hour forecasts and 120 hour forecasts - MAE: 5.7 and 6.87 respectively. Is it usually the case and if it is, I would like to find out the reasons briefly.

I'm fetching forecasts from global models every 6 hours. Forecasts are in the range of 0-144 hours, hourly incremented. I calculated the error metric by using only 6 hour forecasts vs 120 hour forecasts.

Dataset example below:
You can see that models forecast the next 144 hours in every run and I fetch every 6 hours so base\_run\_timestamp increments every 6 hours.
(Base\_run\_timestamp is when the models made their forecasts and forecasted\_timestamp is pretty self-explanatory)

Note: I realized this is a consistent error for my observation point, I applied simple linear regression and the error went straight down to ~1°C so high error is irrelevant.

Thank you!

[![DS_1](https://i.sstatic.net/dkDEM.png)](https://i.sstatic.net/dkDEM.png) [![DS_2](https://i.sstatic.net/I6RA3.png)](https://i.sstatic.net/I6RA3.png)

**Tags:** meteorology, temperature, weather-forecasting, gfs

> **Comments:**
> **username_1**: I think I can answer your question if you provide a little more detail. What is the time step for the different model runs (is it even different?) ? What is the origin of your numbers? A mean average error of 5.7°C for a 6 hour forecast seems far from reasonable to me. Which time stepping method was deployed (this is the most important detail) ?  
> **username_0**: @J.username_1 Sorry for the late response and thanks for your interest. I've updated the question with examples.  
> **username_2**: Sounds like you are probably in an area with significant terrain variation over short distance.  And so you're at a different elevation than your gridpoint?  Hopefully regression can work well often (though I'd expect there may be times it doesn't, times when the weather varies more by elevation)  
> **username_2**: I think otherwise the basic key questions are to try to make sure everything is being done properly... how big a dataset are you using and what data source are using to verify against the forecast.  Also, why are you doing just 6 hr and 120 hr, if you have a full database and are well familiar with this stuff, I would think you could pretty easily code to score all the forecasts and that might point you towards any issues you're having.  
> **username_2**: Because even for a place with a systemic error like poor terrain resolution, the degree of error should change in most situations over that range just because models are poorer at longer range.  And you're also comparing two different times of day (6 hours separate), so even if it were something straightforward like elevation/weather, that shouldn't be as constant for different times of day necessarily.  
> **username_2**: And perhaps explain this linear regression more.  How are you applying that?  To all data points in the forecast using all fhours?  
> **username_0**: @username_2 Thanks for your interest. Observation data comes from national meteorological service stations (sensor data) and the dataset we're using has a time range of 1 year. We've decided to measure our accuracy in different timesteps such as: 6-12-24-36-48-60... until 120 hours, could you elaborate on scoring all the forecasts a bit?  
> **username_2**: ok, that makes sense (would've been a very odd error to have if you were comparing to F0).  And a fine sized sample (I had some concern you might be a student that was trying like one single model run, and of course that wouldn't be representative!).  
> **username_2**: So you're getting this kind of trend for multiple stations (as you said stations)?  That is peculiar.  Perhaps it would be good to compare to reality, we could like at current runs and see if that trend is continuing?  The more results you can offer, the better.  
> **username_2**: I'm not sure what you mean by elaborate on scoring the forecasts, indeed what you sound like you're trying to do makes fine sense.  We do the same thing in the US to a fair degree... though I can rarely dig up the sites... [here was a comparison of GFS MOS temperature errors](https://www.nws.noaa.gov/mdl/synop/gfspara07/warm/00mxmaeovrall.png).  Note that we do create MOS over here, which as far as I understand it tweaks the GFS basic data to account for station details and give a full weather forecast.  
> **username_2**: None of this is stuff I've messed with in a long while, but it is still surprising to have that consistent of an error over many sites.  
> **username_0**: Great, good to learn about MOS and comparison plot was helpful, thanks! We are going with a similar idea actually. Our forecast locations are settlement-based points rather than grids so we're tweaking the nearest grid point forecasts to our location. I'm working with solid meteorology engineers but this is my first time in this domain so I probably need a powerful visualization :) I'll dig deeper into our accuracy results, thanks!  

---

## 0 answers

