  A popular approach for solving zero-sum games is to maintain populations of
policies to approximate the Nash Equilibrium (NE). Previous studies have shown
that Policy Space Response Oracle (PSRO) algorithm is an effective multi-agent
reinforcement learning framework for solving such games. However, repeatedly
training new policies from scratch to approximate Best Response (BR) to
opponents' mixed policies at each iteration is both inefficient and costly.
While some PSRO variants initialize a new policy by inheriting from past BR
policies, this approach limits the exploration of new policies, especially
against challenging opponents. To address this issue, we propose Fusion-PSRO,
which employs policy fusion to initialize policies for better approximation to
BR. By selecting high-quality base policies from meta-NE, policy fusion fuses
the base policies into a new policy through model averaging. This approach
allows the initialized policies to incorporate multiple expert policies, making
it easier to handle difficult opponents compared to inheriting from past BR
policies or initializing from scratch. Moreover, our method only modifies the
policy initialization phase, allowing its application to nearly all PSRO
variants without additional training overhead. Our experiments on
non-transitive matrix games, Leduc Poker, and the more complex Liars Dice
demonstrate that Fusion-PSRO enhances the performance of nearly all PSRO
variants, achieving lower exploitability.
