  The development of intelligent traffic light control systems is essential for
smart transportation management. While some efforts have been made to optimize
the use of individual traffic lights in an isolated way, related studies have
largely ignored the fact that the use of multi-intersection traffic lights is
spatially influenced and there is a temporal dependency of historical traffic
status for current traffic light control. To that end, in this paper, we
propose a novel SpatioTemporal Multi-Agent Reinforcement Learning (STMARL)
framework for effectively capturing the spatio-temporal dependency of multiple
related traffic lights and control these traffic lights in a coordinating way.
Specifically, we first construct the traffic light adjacency graph based on the
spatial structure among traffic lights. Then, historical traffic records will
be integrated with current traffic status via Recurrent Neural Network
structure. Moreover, based on the temporally-dependent traffic information, we
design a Graph Neural Network based model to represent relationships among
multiple traffic lights, and the decision for each traffic light will be made
in a distributed way by the deep Q-learning method. Finally, the experimental
results on both synthetic and real-world data have demonstrated the
effectiveness of our STMARL framework, which also provides an insightful
understanding of the influence mechanism among multi-intersection traffic
lights.
