  When developing a new networking algorithm, it is established practice to run
a randomized experiment, or A/B test, to evaluate its performance. In an A/B
test, traffic is randomly allocated between a treatment group, which uses the
new algorithm, and a control group, which uses the existing algorithm. However,
because networks are congested, both treatment and control traffic compete
against each other for resources in a way that biases the outcome of these
tests. This bias can have a surprisingly large effect; for example, in lab A/B
tests with two widely used congestion control algorithms, the treatment
appeared to deliver 150% higher throughput when used by a few flows, and 75%
lower throughput when used by most flows-despite the fact that the two
algorithms have identical throughput when used by all traffic.
  Beyond the lab, we show that A/B tests can also be biased at scale. In an
experiment run in cooperation with Netflix, estimates from A/B tests mistake
the direction of change of some metrics, miss changes in other metrics, and
overestimate the size of effects. We propose alternative experiment designs,
previously used in online platforms, to more accurately evaluate new algorithms
and allow experimenters to better understand the impact of congestion on their
tests.
