  Making sophisticated, robust, and safe sequential decisions is at the heart
of intelligent systems. This is especially critical for planning in complex
multi-agent environments, where agents need to anticipate other agents'
intentions and possible future actions. Traditional methods formulate the
problem as a Markov Decision Process, but the solutions often rely on various
assumptions and become brittle when presented with corner cases. In contrast,
deep reinforcement learning (Deep RL) has been very effective at finding
policies by simultaneously exploring, interacting, and learning from
environments. Leveraging the powerful Deep RL paradigm, we demonstrate that an
iterative procedure of self-play can create progressively more diverse
environments, leading to the learning of sophisticated and robust multi-agent
policies. We demonstrate this in a challenging multi-agent simulation of
merging traffic, where agents must interact and negotiate with others in order
to successfully merge on or off the road. While the environment starts off
simple, we increase its complexity by iteratively adding an increasingly
diverse set of agents to the agent "zoo" as training progresses. Qualitatively,
we find that through self-play, our policies automatically learn interesting
behaviors such as defensive driving, overtaking, yielding, and the use of
signal lights to communicate intentions to other agents. In addition,
quantitatively, we show a dramatic improvement of the success rate of merging
maneuvers from 63% to over 98%.
