  The increasing prevalence of multi-agent learning systems in society
necessitates understanding how to learn effective and safe policies in
general-sum multi-agent environments against a variety of opponents, including
self-play. General-sum learning is difficult because of non-stationary
opponents and misaligned incentives. Our first main contribution is to show
that many recent approaches to general-sum learning can be derived as
approximations to Stackelberg strategies, which suggests a framework for
developing new multi-agent learning algorithms. We then define non-coincidental
games as games in which the Stackelberg strategy profile is not a Nash
Equilibrium. This notably includes several canonical matrix games and provides
a normative theory for why existing algorithms fail in self-play in such games.
We address this problem by introducing Welfare Equilibria (WE) as a
generalisation of Stackelberg Strategies, which can recover desirable Nash
Equilibria even in non-coincidental games. Finally, we introduce Welfare
Function Search (WelFuSe) as a practical approach to finding desirable WE
against unknown opponents, which finds more mutually desirable solutions in
self-play, while preserving performance against naive learning opponents.
