  In the criminal legal context, risk assessment algorithms are touted as
data-driven, well-tested tools. Studies known as validation tests are typically
cited by practitioners to show that a particular risk assessment algorithm has
predictive accuracy, establishes legitimate differences between risk groups,
and maintains some measure of group fairness in treatment. To establish these
important goals, most tests use a one-shot, single-point measurement. Using a
Polya Urn model, we explore the implication of feedback effects in sequential
scoring-decision processes. We show through simulation that risk can propagate
over sequential decisions in ways that are not captured by one-shot tests. For
example, even a very small or undetectable level of bias in risk allocation can
amplify over sequential risk-based decisions, leading to observable group
differences after a number of decision iterations. Risk assessment tools
operate in a highly complex and path-dependent process, fraught with historical
inequity. We conclude from this study that these tools do not properly account
for compounding effects, and require new approaches to development and
auditing.
