  Online auction scenarios, such as bidding searches on advertising platforms,
often require bidders to participate repeatedly in auctions for identical or
similar items. Most previous studies have only considered the process by which
the seller learns the prior-dependent optimal mechanism in a repeated auction.
However, in this paper, we define a multiagent reinforcement learning
environment in which strategic bidders and the seller learn their strategies
simultaneously and design an automatic bidding algorithm that updates the
strategy of bidders through online interactions. We propose Bid Net to replace
the linear shading function as a representation of the strategic bidders'
strategy, which effectively improves the utility of strategy learned by
bidders. We apply and revise the opponent modeling methods to design the PG
(pseudo-gradient) algorithm, which allows bidders to learn optimal bidding
strategies with predictions of the other agents' strategy transition. We prove
that when a bidder uses the PG algorithm, it can learn the best response to
static opponents. When all bidders adopt the PG algorithm, the system will
converge to the equilibrium of the game induced by the auction. In experiments
with diverse environmental settings and varying opponent strategies, the PG
algorithm maximizes the utility of bidders. We hope that this article will
inspire research on automatic bidding strategies for strategic bidders.
