  In this paper, we propose an offline counterfactual policy estimation
framework called Genie to optimize Sponsored Search Marketplace. Genie employs
an open box simulation engine with click calibration model to compute the KPI
impact of any modification to the system. From the experimental results on Bing
traffic, we showed that Genie performs better than existing observational
approaches that employs randomized experiments for traffic slices that have
frequent policy updates. We also show that Genie can be used to tune completely
new policies efficiently without creating risky randomized experiments due to
cold start problem. As time of today, Genie hosts more than 10000 optimization
jobs yearly which runs more than 30 Million processing node hours of big data
jobs for Bing Ads. For the last 3 years, Genie has been proven to be the one of
the major platforms to optimize Bing Ads Marketplace due to its reliability
under frequent policy changes and its efficiency to minimize risks in real
experiments.
