  We propose a \emph{collaborative} multi-agent reinforcement learning
algorithm named variational policy propagation (VPP) to learn a \emph{joint}
policy through the interactions over agents. We prove that the joint policy is
a Markov Random Field under some mild conditions, which in turn reduces the
policy space effectively. We integrate the variational inference as special
differentiable layers in policy such that the actions can be efficiently
sampled from the Markov Random Field and the overall policy is differentiable.
We evaluate our algorithm on several large scale challenging tasks and
demonstrate that it outperforms previous state-of-the-arts.
