  Real-time bidding (RTB) has become a critical way of online advertising. In
RTB, an advertiser can participate in bidding ad impressions to display its
advertisements. The advertiser determines every impression's bidding price
according to its bidding strategy. Therefore, a good bidding strategy can help
advertisers improve cost efficiency. This paper focuses on optimizing a single
advertiser's bidding strategy using reinforcement learning (RL) in RTB.
Unfortunately, it is challenging to optimize the bidding strategy through RL at
the granularity of impression due to the highly dynamic nature of the RTB
environment. In this paper, we first utilize a widely accepted linear bidding
function to compute every impression's base price and optimize it by a mutable
adjustment factor derived from the RTB auction environment, to avoid optimizing
every impression's bidding price directly. Specifically, we use the maximum
entropy RL algorithm (Soft Actor-Critic) to optimize the adjustment factor
generation policy at the impression-grained level. Finally, the empirical study
on a public dataset demonstrates that the proposed bidding strategy has
superior performance compared with the baselines.
