  Sponsored search is an indispensable business model and a major revenue
contributor of almost all the search engines. From the advertisers' side,
participating in ranking the search results by paying for the sponsored search
advertisement to attract more awareness and purchase facilitates their
commercial goal. From the users' side, presenting personalized advertisement
reflecting their propensity would make their online search experience more
satisfactory. Sponsored search platforms rank the advertisements by a ranking
function to determine the list of advertisements to show and the charging price
for the advertisers. Hence, it is crucial to find a good ranking function which
can simultaneously satisfy the platform, the users and the advertisers.
Moreover, advertisements showing positions under different queries from
different users may associate with advertisement candidates of different bid
price distributions and click probability distributions, which requires the
ranking functions to be optimized adaptively to the traffic characteristics. In
this work, we proposed a generic framework to optimize the ranking functions by
deep reinforcement learning methods. The framework is composed of two parts: an
offline learning part which initializes the ranking functions by learning from
a simulated advertising environment, allowing adequate exploration of the
ranking function parameter space without hurting the performance of the
commercial platform. An online learning part which further optimizes the
ranking functions by adapting to the online data distribution. Experimental
results on a large-scale sponsored search platform confirm the effectiveness of
the proposed method.
