  Human players in professional team sports achieve high level coordination by
dynamically choosing complementary skills and executing primitive actions to
perform these skills. As a step toward creating intelligent agents with this
capability for fully cooperative multi-agent settings, we propose a two-level
hierarchical multi-agent reinforcement learning (MARL) algorithm with
unsupervised skill discovery. Agents learn useful and distinct skills at the
low level via independent Q-learning, while they learn to select complementary
latent skill variables at the high level via centralized multi-agent training
with an extrinsic team reward. The set of low-level skills emerges from an
intrinsic reward that solely promotes the decodability of latent skill
variables from the trajectory of a low-level skill, without the need for
hand-crafted rewards for each skill. For scalable decentralized execution, each
agent independently chooses latent skill variables and primitive actions based
on local observations. Our overall method enables the use of general
cooperative MARL algorithms for training high level policies and single-agent
RL for training low level skills. Experiments on a stochastic high dimensional
team game show the emergence of useful skills and cooperative team play. The
interpretability of the learned skills show the promise of the proposed method
for achieving human-AI cooperation in team sports games.
