  In experimental design, Neyman allocation refers to the practice of
allocating subjects into treated and control groups, potentially in unequal
numbers proportional to their respective standard deviations, with the
objective of minimizing the variance of the treatment effect estimator. This
widely recognized approach increases statistical power in scenarios where the
treated and control groups have different standard deviations, as is often the
case in social experiments, clinical trials, marketing research, and online A/B
testing. However, Neyman allocation cannot be implemented unless the standard
deviations are known in advance. Fortunately, the multi-stage nature of the
aforementioned applications allows the use of earlier stage observations to
estimate the standard deviations, which further guide allocation decisions in
later stages. In this paper, we introduce a competitive analysis framework to
study this multi-stage experimental design problem. We propose a simple
adaptive Neyman allocation algorithm, which almost matches the
information-theoretic limit of conducting experiments. Using online A/B testing
data from a social media site, we demonstrate the effectiveness of our adaptive
Neyman allocation algorithm, highlighting its practicality especially when
applied with only a limited number of stages.
