  Catastrophic regime shifts in complex natural systems may be averted through
advanced detection. Recent work has provided a proof-of-principle that many
systems approaching a catastrophic transition may be identified through the
lens of early warning indicators such as rising variance or increased return
times. Despite widespread appreciation of the difficulties and uncertainty
involved in such forecasts, proposed methods hardly ever characterize their
expected error rates. Without the benefits of replicates, controls, or
hindsight, applications of these approaches must quantify how reliable
different indicators are in avoiding false alarms, and how sensitive they are
to missing subtle warning signs. We propose a model based approach in order to
quantify this trade-off between reliability and sensitivity and allow
comparisons between different indicators. We show these error rates can be
quite severe for common indicators even under favorable assumptions, and also
illustrate how a model-based indicator can improve this performance. We
demonstrate how the performance of an early warning indicator varies in
different data sets, and suggest that uncertainty quantification become a more
central part of early warning predictions.
