  Online experimentation is at the core of Booking.com's customer-centric
product development. While randomised controlled trials are a powerful tool for
estimating the overall effects of product changes on business metrics, they
often fall short in explaining the mechanism of change. This becomes
problematic when decision-making depends on being able to distinguish between
the direct effect of a treatment on some outcome variable and its indirect
effect via a mediator variable. In this paper, we demonstrate the need for
mediation analyses in online experimentation, and use simulated data to show
how these methods help identify and estimate direct causal effect. Failing to
take into account all confounders can lead to biased estimates, so we include
sensitivity analyses to help gauge the robustness of estimates to missing
causal factors.
