  Online recommendation and advertising are two major income channels for
online recommendation platforms (e.g. e-commerce and news feed site). However,
most platforms optimize recommending and advertising strategies by different
teams separately via different techniques, which may lead to suboptimal overall
performances. To this end, in this paper, we propose a novel two-level
reinforcement learning framework to jointly optimize the recommending and
advertising strategies, where the first level generates a list of
recommendations to optimize user experience in the long run; then the second
level inserts ads into the recommendation list that can balance the immediate
advertising revenue from advertisers and the negative influence of ads on
long-term user experience. To be specific, the first level tackles high
combinatorial action space problem that selects a subset items from the large
item space; while the second level determines three internally related tasks,
i.e., (i) whether to insert an ad, and if yes, (ii) the optimal ad and (iii)
the optimal location to insert. The experimental results based on real-world
data demonstrate the effectiveness of the proposed framework. We have released
the implementation code to ease reproductivity.
