  A/B testing, or controlled experiments, is the gold standard approach to
causally compare the performance of algorithms on online platforms. However,
conventional Bernoulli randomization in A/B testing faces many challenges such
as spillover and carryover effects. Our study focuses on another challenge,
especially for A/B testing on two-sided platforms -- budget constraints. Buyers
on two-sided platforms often have limited budgets, where the conventional A/B
testing may be infeasible to be applied, partly because two variants of
allocation algorithms may conflict and lead some buyers to exceed their budgets
if they are implemented simultaneously. We develop a model to describe
two-sided platforms where buyers have limited budgets. We then provide an
optimal experimental design that guarantees small bias and minimum variance.
Bias is lower when there is more budget and a higher supply-demand rate. We
test our experimental design on both synthetic data and real-world data, which
verifies the theoretical results and shows our advantage compared to Bernoulli
randomization.
