  In this study, we apply reinforcement learning techniques and propose what we
call reinforcement mechanism design to tackle the dynamic pricing problem in
sponsored search auctions. In contrast to previous game-theoretical approaches
that heavily rely on rationality and common knowledge among the bidders, we
take a data-driven approach, and try to learn, over repeated interactions, the
set of optimal reserve prices. We implement our approach within the current
sponsored search framework of a major search engine: we first train a buyer
behavior model, via a real bidding data set, that accurately predicts bids
given information that bidders are aware of, including the game parameters
disclosed by the search engine, as well as the bidders' KPI data from previous
rounds. We then put forward a reinforcement/MDP (Markov Decision Process) based
algorithm that optimizes reserve prices over time, in a GSP-like auction. Our
simulations demonstrate that our framework outperforms static optimization
strategies including the ones that are currently in use, as well as several
other dynamic ones.
