  Recent works have applied the Proximal Policy Optimization (PPO) to the
multi-agent cooperative tasks, such as Independent PPO (IPPO); and vanilla
Multi-agent PPO (MAPPO) which has a centralized value function. However,
previous literature shows that MAPPO may not perform as well as Independent PPO
(IPPO) and the Fine-tuned QMIX on Starcraft Multi-Agent Challenge (SMAC).
MAPPO-Feature-Pruned (MAPPO-FP) improves the performance of MAPPO by the
carefully designed agent-specific features, which may be not friendly to
algorithmic utility. By contrast, we find that MAPPO may face the problem of
\textit{The Policies Overfitting in Multi-agent Cooperation(POMAC)}, as they
learn policies by the sampled advantage values. Then POMAC may lead to updating
the multi-agent policies in a suboptimal direction and prevent the agents from
exploring better trajectories. In this paper, to mitigate the multi-agent
policies overfitting, we propose a novel policy regularization method, which
disturbs the advantage values via random Gaussian noise. The experimental
results show that our method outperforms the Fine-tuned QMIX, MAPPO-FP, and
achieves SOTA on SMAC without agent-specific features. We open-source the code
at \url{https://github.com/hijkzzz/noisy-mappo}.
