  The online advertising market, with its thousands of auctions run per second,
presents a daunting challenge for advertisers who wish to optimize their spend
under a budget constraint. Thus, advertising platforms typically provide
automated agents to their customers, which act on their behalf to bid for
impression opportunities in real time at scale. Because these proxy agents are
owned by the platform but use advertiser funds to operate, there is a strong
practical need to balance reliability and explainability of the agent with
optimizing power. We propose a generalizable approach to optimizing bidding
policies in production environments by learning from real data using offline
reinforcement learning. This approach can be used to optimize any
differentiable base policy (practically, a heuristic policy based on principles
which the advertiser can easily understand), and only requires data generated
by the base policy itself. We use a hybrid agent architecture that combines
arbitrary base policies with deep neural networks, where only the optimized
base policy parameters are eventually deployed, and the neural network part is
discarded after training. We demonstrate that such an architecture achieves
statistically significant performance gains in both simulated and at-scale
production bidding environments. Our approach does not incur additional
infrastructure, safety, or explainability costs, as it directly optimizes
parameters of existing production routines without replacing them with black
box-style models like neural networks.
