  Communication lays the foundation for cooperation in human society and in
multi-agent reinforcement learning (MARL). Humans also desire to maintain their
privacy when communicating with others, yet such privacy concern has not been
considered in existing works in MARL. To this end, we propose the
\textit{differentially private multi-agent communication} (DPMAC) algorithm,
which protects the sensitive information of individual agents by equipping each
agent with a local message sender with rigorous $(\epsilon,
\delta)$-differential privacy (DP) guarantee. In contrast to directly
perturbing the messages with predefined DP noise as commonly done in
privacy-preserving scenarios, we adopt a stochastic message sender for each
agent respectively and incorporate the DP requirement into the sender, which
automatically adjusts the learned message distribution to alleviate the
instability caused by DP noise. Further, we prove the existence of a Nash
equilibrium in cooperative MARL with privacy-preserving communication, which
suggests that this problem is game-theoretically learnable. Extensive
experiments demonstrate a clear advantage of DPMAC over baseline methods in
privacy-preserving scenarios.
