<p>Note that weather forecasting systems are near perfect geophysical fluid dynamics. The numerical model for weather is extremely good.  The primary issue is observation as a source of information.</p>
<p>and Yes, new numerical techniques are better every day. but AI works for the physics, not displacing or replacing the physics.</p>

CC BY-SA 4.0
2024-10-16T08:02:32.173Z
I don't think observation can be said as THE primary issue.  Otherwise AI models would struggle the same.  Closure techniques and data assimilation (which isn't the same as the observation themselves) are certainly not perfect even now.  And inability to handle smaller scales still hinders (for instance supercell dynamics can only be somewhat modeled, and those can be important for upscale energy transfer).  Extremely good compared to past, sure.  But still plenty of room to go before hitting any sort of observational resolution incapability/scales where chaos dominates
JeopardyTempest
CC BY-SA 4.0
2024-10-16T08:54:13.257Z
For performance reasons, classical NWP can only use a small fraction of observations.  In a pre-processing step, they perform so-called ["thinning" or "superobbing"](https://www.ecmwf.int/en/elibrary/81345-superobbing-and-finer-thinning-all-sky-humidity-sounder-assimilation).  That's not as bad as it sounds, as two adjacent satellite soundings are of course highly correlated and contain nearly the same information, but ML (being much faster) does not share this restriction.  One hypothesis is that this explains the better results of AIFS (source: personal communication).
gerrit
CC BY-SA 4.0
2024-10-16T14:40:37.330Z
@JeopardyTempest sir. if you had perfect observations in time and space there would be zero requirement for forecasting lol. at the point of observation, you have the solutions for mass, momentum balances and equation of state. The only thing AI can provide from that point is interpolation/extrapolation. The AI is a *thin* layer reducing uncertainty in space, time and scale. Its also worth pointing out how computationally advanced weather forecasting has always been, since the first tide-computing machines. The very definition of 'AI' in weather forecasting is kinda silly from the start.
user156207
CC BY-SA 4.0
2024-10-17T08:33:30.860Z
We don't have even a tiny sliver of the computational power to do the calculations even if we had perfect observations.  Though dynamical models have the purity of working from the actual physics and "computational advanced-ness"... people have can still recognize patterns, and outperform models (although these days with increasingly marginal skill)... yet without using the depth of data or effort... AI works on a very similar basis.
JeopardyTempest
CC BY-SA 4.0
2024-10-17T08:40:44.070Z
The fact that these very early AI models are competitive/perhaps slightly outperforming our best dynamical models suggests we may be in for a long run of AI dominance in meteorological forecasting.  The fundamental equations of course always will have the potential to beat AI because of their perfect foundation on what's happening.  But that's like me walking into a library and saying I can know everything because its all right there.  Thing is I'd have to reach previously far unseen levels of ability to fully realize that potential.  Likewise with dynamical models.
JeopardyTempest
CC BY-SA 4.0
2024-10-15T23:41:04.950Z
2024-10-15T23:41:04.950Z
mfff
<p>Yes, they can, and this is already happening.  This is a rather new and hot topic.</p>
<p>The European Centre for Medium-range Weather Forecasting (ECMWF) is experimentally running <a href="https://www.ecmwf.int/en/newsletter/178/news/aifs-new-ecmwf-forecasting-system" rel="noreferrer">AIFS</a>, a machine-learning based numerical weather prediction (NWP) system.  They developed this after Google and Nvidia, among others, <a href="https://deepmind.google/discover/blog/graphcast-ai-model-for-faster-and-more-accurate-global-weather-forecasting/" rel="noreferrer">showed that this works</a>.  Machine learning based forecasts perform on average better than equation-based NWP systems such as <a href="https://www.ecmwf.int/en/forecasts/documentation-and-support/changes-ecmwf-model" rel="noreferrer">IFS</a>, and it is several orders of magnitude faster.  They are trained using reanalysis data.  Reanalysis is more or less the result of the equation-based forecasting system operated for historical data, except that in that case it's not a forecast but rather an elaborate interpolation to get the sparse, irregular, and sometimes erroneous observations onto a regular quality-controlled 4-dimensional grid for all variables of interest.</p>
<p>Even more experimental is a purely data-driven approach described in <a href="https://arxiv.org/abs/2407.15586" rel="noreferrer">McNally et al. (2024)</a>.  This approach does not need reanalysis but learns directly from past observations: satellite data, ground-based measurements, etc.  It is still experimental and does not currently perform as good as IFS or AIFS, but research is ongoing.</p>
<p>Both systems forecast all weather that NWP-based systems forecast, so that certainly includes hurricanes.  Usually, when they're better at the 500 hPa geopotential height (as compared in initial studies), they're also better at forecasting hurricanes, but studies on specific meteorological phenomena are probably ongoing and not yet published.</p>
<p>The field of machine learning based weather forecasting on a large scale is still relatively new and is a hot topic that is moving very fast.  If you read this answer in 2026 or later, it is probably out of date!</p>

CC BY-SA 4.0
2024-10-15T09:49:02.837Z
Just of note, the better shown on the site isn't in terms of hurricane forecasts particularly, but in terms of large atmospheric patterns (500 mb heights).  Would be interesting to see how it compares statistically to the other hurricane models, though the sites I can thin of it don't include it in the calculations :-/  NOAA's postseason report I expect will
JeopardyTempest
CC BY-SA 4.0
2024-10-15T11:56:25.747Z
@JeopardyTempest I suspect research is ongoing and we'll see studies published on exactly that question.
gerrit
CC BY-SA 4.0
2024-10-16T07:52:06.273Z
Oh certainly.  If I paid attention enough I might know.  I know in the Atlantic AIFS did well with Francine.  And had Milton going south of Tampa earlier than most (though perhaps a tad too far south?)  But that's where numbers are better than one or two hazy recollections :)
JeopardyTempest
CC BY-SA 4.0
2024-10-16T07:54:42.533Z
Crazy world where a pattern recognition system can outperform intense grinding of the fundamental equations of the atmosphere at higher resolution.  Of course you figure you never can know where it will slip up (just as with regular AI), as there's not the fundamental reality behind it (just as how it does answering questions and such), so it's a weird thing to trust in some ways.  But any tool that improves is a good tool!
JeopardyTempest
CC BY-SA 4.0
2024-10-16T09:00:13.150Z
@JeopardyTempest The AI model also uses more data, as it can use all data (NWP needs [superobbing or thinning](https://www.ecmwf.int/en/elibrary/81345-superobbing-and-finer-thinning-all-sky-humidity-sounder-assimilation)) — this might be one reason why it performs better.  But for the reasons you mention, I don't think the physics-based models are going anywhere in the foreseeable future, but will remain running in parallel for comparison in case the AI models predict something funny.
gerrit
CC BY-SA 4.0
2024-10-15T07:47:40.167Z
2024-10-15T12:16:19.387Z
2024-10-15T12:16:19.387Z
gerrit
<p>As a alternative to Weather forecasting systems, Can machine learning prediction algorithms assist in predicting hurricanes?</p>
<p>What will be the input dataset for predicting future hurricanes which will assist machine learning algorithms?</p>

<meteorology><tropical-cyclone><weather-satellites><severe-weather>
Alternative solution to Meterological weather forecasting systems
Prashant Akerkar
# Alternative solution to Meterological weather forecasting systems

(Asked by: username_0 on 2024-10-14)

As a alternative to Weather forecasting systems, Can machine learning prediction algorithms assist in predicting hurricanes?

What will be the input dataset for predicting future hurricanes which will assist machine learning algorithms?

**Tags:** meteorology, tropical-cyclone, weather-satellites, severe-weather


---

## 2 answers

### Answer by username_3 on 2024-10-15. Score: 11 | Accepted answer

Yes, they can, and this is already happening. This is a rather new and hot topic.

The European Centre for Medium-range Weather Forecasting (ECMWF) is experimentally running [AIFS](https://www.ecmwf.int/en/newsletter/178/news/aifs-new-ecmwf-forecasting-system), a machine-learning based numerical weather prediction (NWP) system. They developed this after Google and Nvidia, among others, [showed that this works](https://deepmind.google/discover/blog/graphcast-ai-model-for-faster-and-more-accurate-global-weather-forecasting/). Machine learning based forecasts perform on average better than equation-based NWP systems such as [IFS](https://www.ecmwf.int/en/forecasts/documentation-and-support/changes-ecmwf-model), and it is several orders of magnitude faster. They are trained using reanalysis data. Reanalysis is more or less the result of the equation-based forecasting system operated for historical data, except that in that case it's not a forecast but rather an elaborate interpolation to get the sparse, irregular, and sometimes erroneous observations onto a regular quality-controlled 4-dimensional grid for all variables of interest.

Even more experimental is a purely data-driven approach described in [McNally et al. (2024)](https://arxiv.org/abs/2407.15586). This approach does not need reanalysis but learns directly from past observations: satellite data, ground-based measurements, etc. It is still experimental and does not currently perform as good as IFS or AIFS, but research is ongoing.

Both systems forecast all weather that NWP-based systems forecast, so that certainly includes hurricanes. Usually, when they're better at the 500 hPa geopotential height (as compared in initial studies), they're also better at forecasting hurricanes, but studies on specific meteorological phenomena are probably ongoing and not yet published.

The field of machine learning based weather forecasting on a large scale is still relatively new and is a hot topic that is moving very fast. If you read this answer in 2026 or later, it is probably out of date!

> **Comments:**
> **username_2**: Just of note, the better shown on the site isn't in terms of hurricane forecasts particularly, but in terms of large atmospheric patterns (500 mb heights).  Would be interesting to see how it compares statistically to the other hurricane models, though the sites I can thin of it don't include it in the calculations :-/  NOAA's postseason report I expect will  
> **username_3**: @username_2 I suspect research is ongoing and we'll see studies published on exactly that question.  
> **username_2**: Oh certainly.  If I paid attention enough I might know.  I know in the Atlantic AIFS did well with Francine.  And had Milton going south of Tampa earlier than most (though perhaps a tad too far south?)  But that's where numbers are better than one or two hazy recollections :)  
> **username_2**: Crazy world where a pattern recognition system can outperform intense grinding of the fundamental equations of the atmosphere at higher resolution.  Of course you figure you never can know where it will slip up (just as with regular AI), as there's not the fundamental reality behind it (just as how it does answering questions and such), so it's a weird thing to trust in some ways.  But any tool that improves is a good tool!  
> **username_3**: @username_2 The AI model also uses more data, as it can use all data (NWP needs [superobbing or thinning](https://www.ecmwf.int/en/elibrary/81345-superobbing-and-finer-thinning-all-sky-humidity-sounder-assimilation)) — this might be one reason why it performs better.  But for the reasons you mention, I don't think the physics-based models are going anywhere in the foreseeable future, but will remain running in parallel for comparison in case the AI models predict something funny.  

---

### Answer by username_1 on 2024-10-15. Score: 1

Note that weather forecasting systems are near perfect geophysical fluid dynamics. The numerical model for weather is extremely good. The primary issue is observation as a source of information.

and Yes, new numerical techniques are better every day. but AI works for the physics, not displacing or replacing the physics.

> **Comments:**
> **username_2**: I don't think observation can be said as THE primary issue.  Otherwise AI models would struggle the same.  Closure techniques and data assimilation (which isn't the same as the observation themselves) are certainly not perfect even now.  And inability to handle smaller scales still hinders (for instance supercell dynamics can only be somewhat modeled, and those can be important for upscale energy transfer).  Extremely good compared to past, sure.  But still plenty of room to go before hitting any sort of observational resolution incapability/scales where chaos dominates  
> **username_3**: For performance reasons, classical NWP can only use a small fraction of observations.  In a pre-processing step, they perform so-called ["thinning" or "superobbing"](https://www.ecmwf.int/en/elibrary/81345-superobbing-and-finer-thinning-all-sky-humidity-sounder-assimilation).  That's not as bad as it sounds, as two adjacent satellite soundings are of course highly correlated and contain nearly the same information, but ML (being much faster) does not share this restriction.  One hypothesis is that this explains the better results of AIFS (source: personal communication).  
> **username_4**: @username_2 sir. if you had perfect observations in time and space there would be zero requirement for forecasting lol. at the point of observation, you have the solutions for mass, momentum balances and equation of state. The only thing AI can provide from that point is interpolation/extrapolation. The AI is a *thin* layer reducing uncertainty in space, time and scale. Its also worth pointing out how computationally advanced weather forecasting has always been, since the first tide-computing machines. The very definition of 'AI' in weather forecasting is kinda silly from the start.  
> **username_2**: We don't have even a tiny sliver of the computational power to do the calculations even if we had perfect observations.  Though dynamical models have the purity of working from the actual physics and "computational advanced-ness"... people have can still recognize patterns, and outperform models (although these days with increasingly marginal skill)... yet without using the depth of data or effort... AI works on a very similar basis.  
> **username_2**: The fact that these very early AI models are competitive/perhaps slightly outperforming our best dynamical models suggests we may be in for a long run of AI dominance in meteorological forecasting.  The fundamental equations of course always will have the potential to beat AI because of their perfect foundation on what's happening.  But that's like me walking into a library and saying I can know everything because its all right there.  Thing is I'd have to reach previously far unseen levels of ability to fully realize that potential.  Likewise with dynamical models.  

---

