  Information sharing is key in building team cognition and enables
coordination and cooperation. High-performing human teams also benefit from
acting strategically with hierarchical levels of iterated communication and
rationalizability, meaning a human agent can reason about the actions of their
teammates in their decision-making. Yet, the majority of prior work in
Multi-Agent Reinforcement Learning (MARL) does not support iterated
rationalizability and only encourage inter-agent communication, resulting in a
suboptimal equilibrium cooperation strategy. In this work, we show that
reformulating an agent's policy to be conditional on the policies of its
neighboring teammates inherently maximizes Mutual Information (MI) lower-bound
when optimizing under Policy Gradient (PG). Building on the idea of
decision-making under bounded rationality and cognitive hierarchy theory, we
show that our modified PG approach not only maximizes local agent rewards but
also implicitly reasons about MI between agents without the need for any
explicit ad-hoc regularization terms. Our approach, InfoPG, outperforms
baselines in learning emergent collaborative behaviors and sets the
state-of-the-art in decentralized cooperative MARL tasks. Our experiments
validate the utility of InfoPG by achieving higher sample efficiency and
significantly larger cumulative reward in several complex cooperative
multi-agent domains.
