  Self-Driven Particles (SDP) describe a category of multi-agent systems common
in everyday life, such as flocking birds and traffic flows. In a SDP system,
each agent pursues its own goal and constantly changes its cooperative or
competitive behaviors with its nearby agents. Manually designing the
controllers for such SDP system is time-consuming, while the resulting emergent
behaviors are often not realistic nor generalizable. Thus the realistic
simulation of SDP systems remains challenging. Reinforcement learning provides
an appealing alternative for automating the development of the controller for
SDP. However, previous multi-agent reinforcement learning (MARL) methods define
the agents to be teammates or enemies before hand, which fail to capture the
essence of SDP where the role of each agent varies to be cooperative or
competitive even within one episode. To simulate SDP with MARL, a key challenge
is to coordinate agents' behaviors while still maximizing individual
objectives. Taking traffic simulation as the testing bed, in this work we
develop a novel MARL method called Coordinated Policy Optimization (CoPO),
which incorporates social psychology principle to learn neural controller for
SDP. Experiments show that the proposed method can achieve superior performance
compared to MARL baselines in various metrics. Noticeably the trained vehicles
exhibit complex and diverse social behaviors that improve performance and
safety of the population as a whole. Demo video and source code are available
at: https://decisionforce.github.io/CoPO/
