  Despite the recent advancement in multi-agent reinforcement learning (MARL),
the MARL agents easily overfit the training environment and perform poorly in
the evaluation scenarios where other agents behave differently. Obtaining
generalizable policies for MARL agents is thus necessary but challenging mainly
due to complex multi-agent interactions. In this work, we model the problem
with Markov Games and propose a simple yet effective method, ranked policy
memory (RPM), to collect diverse multi-agent trajectories for training MARL
policies with good generalizability. The main idea of RPM is to maintain a
look-up memory of policies. In particular, we try to acquire various levels of
behaviors by saving policies via ranking the training episode return, i.e., the
episode return of agents in the training environment; when an episode starts,
the learning agent can then choose a policy from the RPM as the behavior
policy. This innovative self-play training framework leverages agents' past
policies and guarantees the diversity of multi-agent interaction in the
training data. We implement RPM on top of MARL algorithms and conduct extensive
experiments on Melting Pot. It has been demonstrated that RPM enables MARL
agents to interact with unseen agents in multi-agent generalization evaluation
scenarios and complete given tasks, and it significantly boosts the performance
up to 402% on average.
