  In multi-agent games, the complexity of the environment can grow
exponentially as the number of agents increases, so it is particularly
challenging to learn good policies when the agent population is large. In this
paper, we introduce Evolutionary Population Curriculum (EPC), a curriculum
learning paradigm that scales up Multi-Agent Reinforcement Learning (MARL) by
progressively increasing the population of training agents in a stage-wise
manner. Furthermore, EPC uses an evolutionary approach to fix an objective
misalignment issue throughout the curriculum: agents successfully trained in an
early stage with a small population are not necessarily the best candidates for
adapting to later stages with scaled populations. Concretely, EPC maintains
multiple sets of agents in each stage, performs mix-and-match and fine-tuning
over these sets and promotes the sets of agents with the best adaptability to
the next stage. We implement EPC on a popular MARL algorithm, MADDPG, and
empirically show that our approach consistently outperforms baselines by a
large margin as the number of agents grows exponentially.
