  In the intrinsically motivated skills acquisition problem, the agent is set
in an environment without any pre-defined goals and needs to acquire an
open-ended repertoire of skills. To do so the agent needs to be autotelic
(deriving from the Greek auto (self) and telos (end goal)): it needs to
generate goals and learn to achieve them following its own intrinsic motivation
rather than external supervision. Autotelic agents have so far been considered
in isolation. But many applications of open-ended learning entail groups of
agents. Multi-agent environments pose an additional challenge for autotelic
agents: to discover and master goals that require cooperation agents must
pursue them simultaneously, but they have low chances of doing so if they
sample them independently. In this work, we propose a new learning paradigm for
modeling such settings, the Decentralized Intrinsically Motivated Skills
Acquisition Problem (Dec-IMSAP), and employ it to solve cooperative navigation
tasks. First, we show that agents setting their goals independently fail to
master the full diversity of goals. Then, we show that a sufficient condition
for achieving this is to ensure that a group aligns its goals, i.e., the agents
pursue the same cooperative goal. Our empirical analysis shows that alignment
enables specialization, an efficient strategy for cooperation. Finally, we
introduce the Goal-coordination game, a fully-decentralized emergent
communication algorithm, where goal alignment emerges from the maximization of
individual rewards in multi-goal cooperative environments and show that it is
able to reach equal performance to a centralized training baseline that
guarantees aligned goals. To our knowledge, this is the first contribution
addressing the problem of intrinsically motivated multi-agent goal exploration
in a decentralized training paradigm.
