  The aim of this paper is to introduce a new design of experiment method for
A/B tests in order to balance the covariate information in all treatment
groups. A/B tests (or "A/B/n tests") refer to the experiments and the
corresponding inference on the treatment effect(s) of a two-level or
multi-level controllable experimental factor. The common practice is to use a
randomized design and perform hypothesis tests on the estimates. However, such
estimation and inference are not always accurate when covariate imbalance
exists among the treatment groups. To overcome this issue, we propose a
discrepancy-based criterion and show that the design minimizing this criterion
significantly improves the accuracy of the treatment effect(s) estimates. The
discrepancy-based criterion is model-free and thus makes the estimation of the
treatment effect(s) robust to the model assumptions. More importantly, the
proposed design is applicable to both continuous and categorical response
measurements. We develop two efficient algorithms to construct the designs by
optimizing the criterion for both offline and online A/B tests. Through
simulation study and a real example, we show that the proposed design approach
achieves good covariate balance and accurate estimation.
