  The forecast accuracy of deep-learning-based weather prediction models is
improving rapidly, leading many to speak of a "second revolution in weather
forecasting". With numerous methods being developed, and limited physical
guarantees offered by deep-learning models, there is a critical need for
comprehensive evaluation of these emerging techniques. While this need has been
partly fulfilled by benchmark datasets, they provide little information on rare
and impactful extreme events, or on compound impact metrics, for which model
accuracy might degrade due to misrepresented dependencies between variables. To
address these issues, we compare deep-learning weather prediction models
(GraphCast, PanguWeather, FourCastNet) and ECMWF's high-resolution forecast
(HRES) system in three case studies: the 2021 Pacific Northwest heatwave, the
2023 South Asian humid heatwave, and the North American winter storm in 2021.
We find evidence that machine learning (ML) weather prediction models can
locally achieve similar accuracy to HRES on record-shattering events such as
the 2021 Pacific Northwest heatwave and even forecast the compound 2021 North
American winter storm substantially better. However, extrapolating to extreme
conditions may impact machine learning models more severely than HRES, as
evidenced by the comparable or superior spatially- and temporally-aggregated
forecast accuracy of HRES for the two heatwaves studied. The ML forecasts also
lack variables required to assess the health risks of events such as the 2023
South Asian humid heatwave. Generally, case-study-driven, impact-centric
evaluation can complement existing research, increase public trust, and aid in
developing reliable ML weather prediction models.
