  Motivated by online display ad exchanges, we study a setting in which an
exchange repeatedly interacts with bidders who have quota, making decisions
about which subsets of bidders are called to participate in ad-slot-specific
auctions. A bidder with quota cannot respond to more than a certain number of
calls per second. In practice, random throttling is the principal solution by
which these constraints are enforced. Given the repeated nature of the
interaction with its bidders, the exchange has access to data containing
information about each bidder's segments of interest. This information can be
utilized to design smarter callout mechanisms --- with the potential of
improving the exchange's long-term revenue. In this work, we present a general
framework for evaluating and comparing the performance of various callout
mechanisms using historical auction data only. To measure the impact of a
callout mechanism on long-term revenue, we propose a strategic model that
captures the repeated interaction between the exchange and bidders. Our model
leads us to two metrics for performance: immediate revenue impact and social
welfare. Next we present an empirical framework for estimating these two
metrics from historical data. For the baseline to compare against, we consider
random throttling, as well as a greedy algorithm with certain theoretical
guarantees. We propose several natural callout mechanisms and investigate them
through our framework on both synthetic and real auction data. We characterize
the conditions under which each heuristic performs well and show that, in
addition to being computationally faster, in practice our heuristics
consistently and significantly outperform the baselines.
