  To increase brand awareness, many advertisers conclude contracts with
advertising platforms to purchase traffic and then deliver advertisements to
target audiences. In a whole delivery period, advertisers usually desire a
certain impression count for the ads, and they also expect that the delivery
performance is as good as possible (e.g., obtaining high click-through rate).
Advertising platforms employ pacing algorithms to satisfy the demands via
adjusting the selection probabilities to traffic requests in real-time.
However, the delivery procedure is also affected by the strategies from
publishers, which cannot be controlled by advertising platforms. Preloading is
a widely used strategy for many types of ads (e.g., video ads) to make sure
that the response time for displaying after a traffic request is legitimate,
which results in delayed impression phenomenon. Traditional pacing algorithms
cannot handle the preloading nature well because they rely on immediate
feedback signals, and may fail to guarantee the demands from advertisers.
  In this paper, we focus on a new research problem of impression pacing for
preloaded ads, and propose a Reinforcement Learning To Pace framework RLTP. It
learns a pacing agent that sequentially produces selection probabilities in the
whole delivery period. To jointly optimize the two objectives of impression
count and delivery performance, RLTP employs tailored reward estimator to
satisfy the guaranteed impression count, penalize the over-delivery and
maximize the traffic value. Experiments on large-scale industrial datasets
verify that RLTP outperforms baseline pacing algorithms by a large margin. We
have deployed the RLTP framework online to our advertising platform, and
results show that it achieves significant uplift to core metrics including
delivery completion rate and click-through rate.
