  We study the role of intrinsic motivation as an exploration bias for
reinforcement learning in sparse-reward synergistic tasks, which are tasks
where multiple agents must work together to achieve a goal they could not
individually. Our key idea is that a good guiding principle for intrinsic
motivation in synergistic tasks is to take actions which affect the world in
ways that would not be achieved if the agents were acting on their own. Thus,
we propose to incentivize agents to take (joint) actions whose effects cannot
be predicted via a composition of the predicted effect for each individual
agent. We study two instantiations of this idea, one based on the true states
encountered, and another based on a dynamics model trained concurrently with
the policy. While the former is simpler, the latter has the benefit of being
analytically differentiable with respect to the action taken. We validate our
approach in robotic bimanual manipulation and multi-agent locomotion tasks with
sparse rewards; we find that our approach yields more efficient learning than
both 1) training with only the sparse reward and 2) using the typical
surprise-based formulation of intrinsic motivation, which does not bias toward
synergistic behavior. Videos are available on the project webpage:
https://sites.google.com/view/iclr2020-synergistic.
