  Recently, auto-bidding technique has become an essential tool to increase the
revenue of advertisers. Facing the complex and ever-changing bidding
environments in the real-world advertising system (RAS), state-of-the-art
auto-bidding policies usually leverage reinforcement learning (RL) algorithms
to generate real-time bids on behalf of the advertisers. Due to safety
concerns, it was believed that the RL training process can only be carried out
in an offline virtual advertising system (VAS) that is built based on the
historical data generated in the RAS. In this paper, we argue that there exists
significant gaps between the VAS and RAS, making the RL training process suffer
from the problem of inconsistency between online and offline (IBOO). Firstly,
we formally define the IBOO and systematically analyze its causes and
influences. Then, to avoid the IBOO, we propose a sustainable online RL (SORL)
framework that trains the auto-bidding policy by directly interacting with the
RAS, instead of learning in the VAS. Specifically, based on our proof of the
Lipschitz smooth property of the Q function, we design a safe and efficient
online exploration (SER) policy for continuously collecting data from the RAS.
Meanwhile, we derive the theoretical lower bound on the safety of the SER
policy. We also develop a variance-suppressed conservative Q-learning (V-CQL)
method to effectively and stably learn the auto-bidding policy with the
collected data. Finally, extensive simulated and real-world experiments
validate the superiority of our approach over the state-of-the-art auto-bidding
algorithm.
