  In online display advertising, guaranteed contracts and real-time bidding
(RTB) are two major ways to sell impressions for a publisher. Despite the
increasing popularity of RTB, there is still half of online display advertising
revenue generated from guaranteed contracts. Therefore, simultaneously selling
impressions through both guaranteed contracts and RTB is a straightforward
choice for a publisher to maximize its yield. However, deriving the optimal
strategy to allocate impressions is not a trivial task, especially when the
environment is unstable in real-world applications. In this paper, we formulate
the impression allocation problem as an auction problem where each contract can
submit virtual bids for individual impressions. With this formulation, we
derive the optimal impression allocation strategy by solving the optimal
bidding functions for contracts. Since the bids from contracts are decided by
the publisher, we propose a multi-agent reinforcement learning (MARL) approach
to derive cooperative policies for the publisher to maximize its yield in an
unstable environment. The proposed approach also resolves the common challenges
in MARL such as input dimension explosion, reward credit assignment, and
non-stationary environment. Experimental evaluations on large-scale real
datasets demonstrate the effectiveness of our approach.
