  Buyers (e.g., advertisers) often have limited financial and processing
resources, and so their participation in auctions is throttled. Changes to
auctions may affect bids or throttling and any change may affect what winners
pay. This paper shows that if an A/B experiment affects only bids, then the
observed treatment effect is unbiased when all the bidders in an auction are
randomly assigned to A or B but it can be severely biased otherwise, even in
the absence of throttling. Experiments that affect throttling algorithms can
also be badly biased, but the bias can be substantially reduced if the budget
for each advertiser in the experiment is allocated to separate pots for the A
and B arms of the experiment.
