  For many application areas A/B testing, which partitions users of a system
into an A (control) and B (treatment) group to experiment between several
application designs, enables Internet companies to optimize their services to
the behavioral patterns of their users. Unfortunately, the A/B testing
framework cannot be applied in a straightforward manner to applications like
auctions where the users (a.k.a., bidders) submit bids before the partitioning
into the A and B groups is made. This paper combines auction theoretic modeling
with the A/B testing framework to develop methodology for A/B testing auctions.
The accuracy of our method %, assuming the auction is directly comparable to
ideal A/B testing where there is no interference between A and B. Our results
are based on an extension and improved analysis of the inference method of
Chawla et al. (2014).
