  A/B testing is an important decision making tool in product development
because can provide an accurate estimate of the average treatment effect of a
new features, which allows developers to understand how the business impact of
new changes to products or algorithms. However, an important assumption of A/B
testing, Stable Unit Treatment Value Assumption (SUTVA), is not always a valid
assumption to make, especially for products that facilitate interactions
between individuals. In contexts like one-to-one messaging we should expect
network interference; if an experimental manipulation is effective, behavior of
the treatment group is likely to influence members in the control group by
sending them messages, violating this assumption. In this paper, we propose a
novel method that can be used to account for network effects when A/B testing
changes to one-to-one interactions. Our method is an edge-based analysis that
can be applied to standard Bernoulli randomized experiments to retrieve an
average treatment effect that is not influenced by network interference. We
develop a theoretical model, and methods for computing point estimates and
variances of effects of interest via network-consistent permutation testing. We
then apply our technique to real data from experiments conducted on the
messaging product at LinkedIn. We find empirical support for our model, and
evidence that the standard method of analysis for A/B tests underestimates the
impact of new features in one-to-one messaging contexts.
