  Early warning systems (EWS) are predictive tools at the center of recent
efforts to improve graduation rates in public schools across the United States.
These systems assist in targeting interventions to individual students by
predicting which students are at risk of dropping out. Despite significant
investments in their widespread adoption, there remain large gaps in our
understanding of the efficacy of EWS, and the role of statistical risk scores
in education.
  In this work, we draw on nearly a decade's worth of data from a system used
throughout Wisconsin to provide the first large-scale evaluation of the
long-term impact of EWS on graduation outcomes. We present empirical evidence
that the prediction system accurately sorts students by their dropout risk. We
also find that it may have caused a single-digit percentage increase in
graduation rates, though our empirical analyses cannot reliably rule out that
there has been no positive treatment effect.
  Going beyond a retrospective evaluation of DEWS, we draw attention to a
central question at the heart of the use of EWS: Are individual risk scores
necessary for effectively targeting interventions? We propose a simple
mechanism that only uses information about students' environments -- such as
their schools, and districts -- and argue that this mechanism can target
interventions just as efficiently as the individual risk score-based mechanism.
Our argument holds even if individual predictions are highly accurate and
effective interventions exist. In addition to motivating this simple targeting
mechanism, our work provides a novel empirical backbone for the robust
qualitative understanding among education researchers that dropout is
structurally determined. Combined, our insights call into question the marginal
value of individual predictions in settings where outcomes are driven by high
levels of inequality.
