  The importance of Facebook advertising has risen dramatically in recent
years, with the platform accounting for almost 20% of the global online ad
spend in 2017. An important consideration in advertising is incrementality: how
much of the change in an experimental metric is an advertising campaign
responsible for. To measure incrementality, Facebook provide lift studies. As
Facebook lift studies differ from standard A/B tests, the online
experimentation literature does not describe how to calculate parameters such
as power and minimum sample size. Facebook also offer multi-cell lift tests,
which can be used to compare campaigns that don't have statistically identical
audiences. In this case, there is no literature describing how to measure the
significance of the difference in incrementality between cells, or how to
estimate the power or minimum sample size. We fill these gaps in the literature
by providing the statistical power and required sample size calculation for
Facebook lift studies. We then generalise the statistical significance, power,
and required sample size calculation to multi-cell lift studies. We represent
our results theoretically in terms of the distributions of test metrics and in
practical terms relating to the metrics used by practitioners, making all of
our code publicly available.
