  In designing an online A/B experiment, it is crucial to select a sample size
and duration that ensure the resulting confidence interval (CI) for the
treatment effect is the right width to detect an effect of meaningful magnitude
with sufficient statistical power without wasting resources. While the
relationship between sample size and CI width is well understood, the effect of
experiment duration on CI width remains less clear. This paper provides an
analytical formula for the width of a CI based on a ratio treatment effect
estimator as a function of both sample size (N) and duration (T). The formula
is derived from a mixed effects model with two variance components. One
component, referred to as the temporal variance, persists over time for
experiments where the same users are kept in the same experiment arm across
different days. The remaining error variance component, by contrast, decays to
zero as T gets large. The formula we derive introduces a key parameter that we
call the user-specific temporal correlation (UTC), which quantifies the
relative sizes of the two variance components and can be estimated from
historical experiments. Higher UTC indicates a slower decay in CI width over
time. On the other hand, when the UTC is 0 -- as for experiments where users
shuffle in and out of the experiment across days -- the CI width decays at the
standard parametric 1/T rate. We also study how access to pre-period data for
the users in the experiment affects the CI width decay. We show our formula
closely explains CI widths on real A/B experiments at YouTube.
