  Technology companies are increasingly using randomized controlled trials
(RCTs) as part of their development process. Despite having fine control over
engineering systems and data instrumentation, these RCTs can still be
imperfectly executed. In fact, online experimentation suffers from many of the
same biases seen in biomedical RCTs including opt-in and user activity bias,
selection bias, non-compliance with the treatment, and more generally,
challenges in the ability to test the question of interest. The result of these
imperfections can lead to a bias in the estimated causal effect, a loss in
statistical power, an attenuation of the effect, or even a need to reframe the
question that can be answered. This paper aims to make practitioners of
experimentation more aware of imperfections in technology-industry RCTs, which
can be hidden throughout the engineering stack or in the design process.
