  Although reinforcement learning (RL) is considered the gold standard for
policy design, it may not always provide a robust solution in various
scenarios. This can result in severe performance degradation when the
environment is exposed to potential disturbances. Adversarial training using a
two-player max-min game has been proven effective in enhancing the robustness
of RL agents. In this work, we extend the two-player game by introducing an
adversarial herd, which involves a group of adversaries, in order to address
($\textit{i}$) the difficulty of the inner optimization problem, and
($\textit{ii}$) the potential over pessimism caused by the selection of a
candidate adversary set that may include unlikely scenarios. We first prove
that adversarial herds can efficiently approximate the inner optimization
problem. Then we address the second issue by replacing the worst-case
performance in the inner optimization with the average performance over the
worst-$k$ adversaries. We evaluate the proposed method on multiple MuJoCo
environments. Experimental results demonstrate that our approach consistently
generates more robust policies.
