  Accurate prediction of climate in the subseasonal-to-seasonal scale is
crucial for disaster preparedness and robust decision making amidst climate
change. Yet, forecasting beyond the weather timescale is challenging because it
deals with problems other than initial conditions, including boundary
interaction, butterfly effect, and our inherent lack of physical understanding.
At present, existing benchmarks tend to have shorter forecasting range of up-to
15 days, do not include a wide range of operational baselines, and lack
physics-based constraints for explainability. Thus, we propose ChaosBench, a
challenging benchmark to extend the predictability range of data-driven weather
emulators to S2S timescale. First, ChaosBench is comprised of variables beyond
the typical surface-atmospheric ERA5 to also include ocean, ice, and land
reanalysis products that span over 45 years to allow for full Earth system
emulation that respects boundary conditions. We also propose physics-based, in
addition to deterministic and probabilistic metrics, to ensure a
physically-consistent ensemble that accounts for butterfly effect. Furthermore,
we evaluate on a diverse set of physics-based forecasts from four national
weather agencies as baselines to our data-driven counterpart such as ClimaX,
PanguWeather, GraphCast, and FourCastNetV2. Overall, we find methods originally
developed for weather-scale applications fail on S2S task: their performance
simply collapse to an unskilled climatology. Nonetheless, we outline and
demonstrate several strategies that can potentially extend the predictability
range of existing weather emulators, including the use of ensembles and robust
control of error propagation. Our benchmark, datasets, and instructions are
available at https://leap-stc.github.io/ChaosBench.
