  Policy-Space Response Oracles (PSRO) is a general algorithmic framework for
learning policies in multiagent systems by interleaving empirical game analysis
with deep reinforcement learning (Deep RL). At each iteration, Deep RL is
invoked to train a best response to a mixture of opponent policies. The
repeated application of Deep RL poses an expensive computational burden as we
look to apply this algorithm to more complex domains. We introduce two
variations of PSRO designed to reduce the amount of simulation required during
Deep RL training. Both algorithms modify how PSRO adds new policies to the
empirical game, based on learned responses to a single opponent policy. The
first, Mixed-Oracles, transfers knowledge from previous iterations of Deep RL,
requiring training only against the opponent's newest policy. The second,
Mixed-Opponents, constructs a pure-strategy opponent by mixing existing
strategy's action-value estimates, instead of their policies. Learning against
a single policy mitigates variance in state outcomes that is induced by an
unobserved distribution of opponents. We empirically demonstrate that these
algorithms substantially reduce the amount of simulation during training
required by PSRO, while producing equivalent or better solutions to the game.
