  Autocurricular training is an important sub-area of multi-agent reinforcement
learning~(MARL) that allows multiple agents to learn emergent skills in an
unsupervised co-evolving scheme. The robotics community has experimented
autocurricular training with physically grounded problems, such as robust
control and interactive manipulation tasks. However, the asymmetric nature of
these tasks makes the generation of sophisticated policies challenging. Indeed,
the asymmetry in the environment may implicitly or explicitly provide an
advantage to a subset of agents which could, in turn, lead to a low-quality
equilibrium. This paper proposes a novel game-theoretic algorithm, Stackelberg
Multi-Agent Deep Deterministic Policy Gradient (ST-MADDPG), which formulates a
two-player MARL problem as a Stackelberg game with one player as the `leader'
and the other as the `follower' in a hierarchical interaction structure wherein
the leader has an advantage. We first demonstrate that the leader's advantage
from ST-MADDPG can be used to alleviate the inherent asymmetry in the
environment. By exploiting the leader's advantage, ST-MADDPG improves the
quality of a co-evolution process and results in more sophisticated and complex
strategies that work well even against an unseen strong opponent.
