  Reinforcement learning (RL) is an effective technique for training
decision-making agents through interactions with their environment. The advent
of deep learning has been associated with highly notable successes with
sequential decision making problems - such as defeating some of the
highest-ranked human players at Go. In digital advertising, real-time bidding
(RTB) is a common method of allocating advertising inventory through real-time
auctions. Bidding strategies need to incorporate logic for dynamically
adjusting parameters in order to deliver pre-assigned campaign goals. Here we
discuss techniques toward using RL to train bidding agents. As a campaign
metric we particularly focused on viewability: the percentage of inventory
which goes on to be viewed by an end user.
  This paper is presented as a survey of techniques and experiments which we
developed through the course of this research. We discuss expanding our
training data to include edge cases by training on simulated interactions. We
discuss the experimental results comparing the performance of several promising
RL algorithms, and an approach to hyperparameter optimization of an
actor/critic training pipeline through Bayesian optimization. Finally, we
present live-traffic tests of some of our RL agents against a rule-based
feedback-control approach, demonstrating the potential for this method as well
as areas for further improvement. This paper therefore presents an arrangement
of our findings in this quickly developing field, and ways that it can be
applied to an RTB use case.
