  Many organizations utilize large-scale online controlled experiments (OCEs)
to accelerate innovation. Having high statistical power to detect small
differences between control and treatment accurately is critical, as even small
changes in key metrics can be worth millions of dollars or indicate user
dissatisfaction for a very large number of users. For large-scale OCE, the
duration is typically short (e.g. two weeks) to expedite changes and
improvements to the product. In this paper, we examine two common approaches
for analyzing usage data collected from users within the time window of an
experiment, which can differ in accuracy and power. The open approach includes
all relevant usage data from all active users for the entire duration of the
experiment. The bounded approach includes data from a fixed period of
observation for each user (e.g. seven days after exposure) after the first time
a user became active in the experiment window.
