  We study multi-agent reinforcement learning (MARL) with centralized training
and decentralized execution. During the training, new agents may join, and
existing agents may unexpectedly leave the training. In such situations, a
standard deep MARL model must be trained again from scratch, which is very
time-consuming. To tackle this problem, we propose a special network
architecture with a few-shot learning algorithm that allows the number of
agents to vary during centralized training. In particular, when a new agent
joins the centralized training, our few-shot learning algorithm trains its
policy network and value network using a small number of samples; when an agent
leaves the training, the training process of the remaining agents is not
affected. Our experiments show that using the proposed network architecture and
algorithm, model adaptation when new agents join can be 100+ times faster than
the baseline. Our work is applicable to any setting, including cooperative,
competitive, and mixed.
