  In multi-agent reinforcement learning, discovering successful collective
behaviors is challenging as it requires exploring a joint action space that
grows exponentially with the number of agents. While the tractability of
independent agent-wise exploration is appealing, this approach fails on tasks
that require elaborate group strategies. We argue that coordinating the agents'
policies can guide their exploration and we investigate techniques to promote
such an inductive bias. We propose two policy regularization methods: TeamReg,
which is based on inter-agent action predictability and CoachReg that relies on
synchronized behavior selection. We evaluate each approach on four challenging
continuous control tasks with sparse rewards that require varying levels of
coordination as well as on the discrete action Google Research Football
environment. Our experiments show improved performance across many cooperative
multi-agent problems. Finally, we analyze the effects of our proposed methods
on the policies that our agents learn and show that our methods successfully
enforce the qualities that we propose as proxies for coordinated behaviors.
