  Traffic signal control (TSC) is a promising low-cost measure to enhance
transportation efficiency without affecting existing road infrastructure. While
various reinforcement learning-based TSC methods have been proposed and
experimentally outperform conventional rule-based methods, none of them has
been deployed in the real world. An essential gap lies in the
oversimplification of the scenarios in terms of intersection heterogeneity and
road network intricacy. To make TSC applicable in urban traffic management, we
target TSC coordination in city-scale high-authenticity road networks, aiming
to solve the three unique and important challenges: city-level scalability,
heterogeneity of real-world intersections, and effective coordination among
intricate neighbor connections. Since optimizing multiple agents in a
parameter-sharing paradigm can boost the training efficiency and help achieve
scalability, we propose our method, CityLight, based on the well-acknowledged
optimization framework, parameter-sharing MAPPO. To ensure the unified policy
network can learn to fit large-scale heterogeneous intersections and tackle the
intricate between-neighbor coordination, CityLight proposes a universal
representation module that consists of two key designs: heterogeneous
intersection alignment and neighborhood impact alignment for coordination. To
further boost coordination, CityLight adopts neighborhood-integrated rewards to
transition from achieving local optimal to global optimal. Extensive
experiments on datasets with hundreds to tens of thousands of real-world
intersections and authentic traffic demands validate the surprising
effectiveness and generalizability of CityLight, with an overall performance
gain of 11.66% and a 22.59% improvement in transfer scenarios in terms of
throughput.
