  Surrogate index approaches have recently become a popular method of
estimating longer-term impact from shorter-term outcomes. In this paper, we
leverage 1098 test arms from 200 A/B tests at Netflix to empirically
investigate to what degree would decisions made using a surrogate index
utilizing 14 days of data would align with those made using direct measurement
of day 63 treatment effects. Focusing specifically on linear "auto-surrogate"
models that utilize the shorter-term observations of the long-term outcome of
interest, we find that the statistical inferences that we would draw from using
the surrogate index are ~95% consistent with those from directly measuring the
long-term treatment effect. Moreover, when we restrict ourselves to the set of
tests that would be "launched" (i.e. positive and statistically significant)
based on the 63-day directly measured treatment effects, we find that relying
instead on the surrogate index achieves 79% and 65% recall.
