  Chaos and unpredictability are traditionally synonymous, yet large-scale
machine learning methods recently have demonstrated a surprising ability to
forecast chaotic systems well beyond typical predictability horizons. However,
recent works disagree on whether specialized methods grounded in dynamical
systems theory, such as reservoir computers or neural ordinary differential
equations, outperform general-purpose large-scale learning methods such as
transformers or recurrent neural networks. These prior studies perform
comparisons on few individually-chosen chaotic systems, thereby precluding
robust quantification of how statistical modeling choices and dynamical
invariants of different chaotic systems jointly determine empirical
predictability. Here, we perform the largest to-date comparative study of
forecasting methods on the classical problem of forecasting chaos: we benchmark
24 state-of-the-art forecasting methods on a crowdsourced database of 135
low-dimensional systems with 17 forecast metrics. We find that large-scale,
domain-agnostic forecasting methods consistently produce predictions that
remain accurate up to two dozen Lyapunov times, thereby accessing a new
long-horizon forecasting regime well beyond classical methods. We find that, in
this regime, accuracy decorrelates with classical invariant measures of
predictability like the Lyapunov exponent. However, in data-limited settings
outside the long-horizon regime, we find that physics-based hybrid methods
retain a comparative advantage due to their strong inductive biases.
