  Multi-agent reinforcement learning (MARL) is a promising framework for
solving complex tasks with many agents. However, a key challenge in MARL is
defining private utility functions that ensure coordination when training
decentralized agents. This challenge is especially prevalent in unstructured
tasks with sparse rewards and many agents. We show that successor features can
help address this challenge by disentangling an individual agent's impact on
the global value function from that of all other agents. We use this
disentanglement to compactly represent private utilities that support stable
training of decentralized agents in unstructured tasks. We implement our
approach using a centralized training, decentralized execution architecture and
test it in a variety of multi-agent environments. Our results show improved
performance and training time relative to existing methods and suggest that
disentanglement of successor features offers a promising approach to
coordination in MARL.
