  Solving Nash equilibrium is the key challenge in normal-form games with large
strategy spaces, where open-ended learning frameworks offer an efficient
approach. In this work, we propose an innovative unified open-ended learning
framework A-PSRO, i.e., Advantage Policy Space Response Oracle, as a
comprehensive framework for both zero-sum and general-sum games. In particular,
we introduce the advantage function as an enhanced evaluation metric for
strategies, enabling a unified learning objective for agents engaged in
normal-form games. We prove that the advantage function exhibits favorable
properties and is connected with the Nash equilibrium, which can be used as an
objective to guide agents to learn strategies efficiently. Our experiments
reveal that A-PSRO achieves a considerable decrease in exploitability in
zero-sum games and an escalation in rewards in general-sum games, significantly
outperforming previous PSRO algorithms.
