  Data-driven weather prediction models (DDWPs) have made rapid strides in
recent years, demonstrating an ability to approximate Numerical Weather
Prediction (NWP) models to a high degree of accuracy. The fast, accurate, and
low-cost DDWP forecasts make their use in operational forecasting an attractive
proposition, however, there remains work to be done in rigorously evaluating
DDWPs in a true operational setting. Typically trained and evaluated using ERA5
reanalysis data, DDWPs have been tested only in a simulation, which cannot
represent the real world with complete accuracy even if it is of a very high
quality. The safe use of DDWPs in operational forecasting requires more
thorough "real-world" verification, as well as a careful examination of how
DDWPs are currently trained and evaluated. It is worth asking, for instance,
how well do the reanalysis datasets, used for training, simulate the real
world? With an eye towards climate justice and the uneven availability of
weather data: is the simulation equally good for all regions of the world, and
would DDWPs exacerbate biases present in the training data? Does a good
performance in simulation correspond to good performance in operational
settings? In addition to approximating the physics of NWP models, how can ML be
uniquely deployed to provide more accurate weather forecasts? As a first step
towards answering such questions, we present a robust dataset of in-situ
observations derived from the NOAA MADIS program to serve as a benchmark to
validate DDWPs in an operational setting. By providing a large corpus of
quality-controlled, in-situ observations, this dataset provides a meaningful
real-world task that all NWPs and DDWPs can be tested against. We hope that
this data can be used not only to rigorously and fairly compare operational
weather models but also to spur future research in new directions.
