  Many recent works have turned to multi-agent reinforcement learning (MARL)
for adaptive traffic signal control to optimize the travel time of vehicles
over large urban networks. However, achieving effective and scalable
cooperation among junctions (agents) remains an open challenge, as existing
methods often rely on extensive, non-generalizable reward shaping or on
non-scalable centralized learning. To address these problems, we propose a new
MARL method for traffic signal control, SocialLight, which learns cooperative
traffic control policies by distributedly estimating the individual marginal
contribution of agents on their local neighborhood. SocialLight relies on the
Asynchronous Actor Critic (A3C) framework, and makes learning scalable by
learning a locally-centralized critic conditioned over the states and actions
of neighboring agents, used by agents to estimate individual contributions by
counterfactual reasoning. We further introduce important modifications to the
advantage calculation that help stabilize policy updates. These modifications
decouple the impact of the neighbors' actions on the computed advantages,
thereby reducing the variance in the gradient updates. We benchmark our trained
network against state-of-the-art traffic signal control methods on standard
benchmarks in two traffic simulators, SUMO and CityFlow. Our results show that
SocialLight exhibits improved scalability to larger road networks and better
performance across usual traffic metrics.
