  Reinforcement Learning (RL) is an effective tool for controller design but
can struggle with issues of robustness, failing catastrophically when the
underlying system dynamics are perturbed. The Robust RL formulation tackles
this by adding worst-case adversarial noise to the dynamics and constructing
the noise distribution as the solution to a zero-sum minimax game. However,
existing work on learning solutions to the Robust RL formulation has primarily
focused on training a single RL agent against a single adversary. In this work,
we demonstrate that using a single adversary does not consistently yield
robustness to dynamics variations under standard parametrizations of the
adversary; the resulting policy is highly exploitable by new adversaries. We
propose a population-based augmentation to the Robust RL formulation in which
we randomly initialize a population of adversaries and sample from the
population uniformly during training. We empirically validate across robotics
benchmarks that the use of an adversarial population results in a more robust
policy that also improves out-of-distribution generalization. Finally, we
demonstrate that this approach provides comparable robustness and
generalization as domain randomization on these benchmarks while avoiding a
ubiquitous domain randomization failure mode.
