  Discovering successful coordinated behaviors is a central challenge in
Multi-Agent Reinforcement Learning (MARL) since it requires exploring a joint
action space that grows exponentially with the number of agents. In this paper,
we propose a mechanism for achieving sufficient exploration and coordination in
a team of agents. Specifically, agents are rewarded for contributing to a more
diversified team behavior by employing proper intrinsic motivation functions.
To learn meaningful coordination protocols, we structure agents' interactions
by introducing a novel framework, where at each timestep, an agent simulates
counterfactual rollouts of its policy and, through a sequence of computations,
assesses the gap between other agents' current behaviors and their targets.
Actions that minimize the gap are considered highly influential and are
rewarded. We evaluate our approach on a set of challenging tasks with sparse
rewards and partial observability that require learning complex cooperative
strategies under a proper exploration scheme, such as the StarCraft Multi-Agent
Challenge. Our methods show significantly improved performances over different
baselines across all tasks.
