  Self-play (SP) is a popular multi-agent reinforcement learning (MARL)
framework for solving competitive games, where each agent optimizes policy by
treating others as part of the environment. Despite the empirical successes,
the theoretical properties of SP-based methods are limited to two-player
zero-sum games. However, for mixed cooperative-competitive games where agents
on the same team need to cooperate with each other, we can show a simple
counter-example where SP-based methods cannot converge to a global Nash
equilibrium (NE) with high probability. Alternatively, Policy-Space Response
Oracles (PSRO) is an iterative framework for learning NE, where the best
responses w.r.t. previous policies are learned in each iteration. PSRO can be
directly extended to mixed cooperative-competitive settings by jointly learning
team best responses with all convergence properties unchanged. However, PSRO
requires repeatedly training joint policies from scratch till convergence,
which makes it hard to scale to complex games. In this work, we develop a novel
algorithm, Fictitious Cross-Play (FXP), which inherits the benefits from both
frameworks. FXP simultaneously trains an SP-based main policy and a counter
population of best response policies. The main policy is trained by fictitious
self-play and cross-play against the counter population, while the counter
policies are trained as the best responses to the main policy's past versions.
We validate our method in matrix games and show that FXP converges to global
NEs while SP methods fail. We also conduct experiments in a gridworld domain,
where FXP achieves higher Elo ratings and lower exploitabilities than
baselines, and a more challenging football game, where FXP defeats SOTA models
with over 94% win rate.
