  The goal of this work is to provide a viable solution based on reinforcement
learning for traffic signal control problems. Although the state-of-the-art
reinforcement learning approaches have yielded great success in a variety of
domains, directly applying it to alleviate traffic congestion can be
challenging, considering the requirement of high sample efficiency and how
training data is gathered. In this work, we address several challenges that we
encountered when we attempted to mitigate serious traffic congestion occurring
in a metropolitan area. Specifically, we are required to provide a solution
that is able to (1) handle the traffic signal control when certain surveillance
cameras that retrieve information for reinforcement learning are down, (2)
learn from batch data without a traffic simulator, and (3) make control
decisions without shared information across intersections. We present a
two-stage framework to deal with the above-mentioned situations. The framework
can be decomposed into an Evolution Strategies approach that gives a fixed-time
traffic signal control schedule and a multi-agent off-policy reinforcement
learning that is capable of learning from batch data with the aid of three
proposed components, bounded action, batch augmentation, and surrogate reward
clipping. Our experiments show that the proposed framework reduces traffic
congestion by 36% in terms of waiting time compared with the currently used
fixed-time traffic signal plan. Furthermore, the framework requires only 600
queries to a simulator to achieve the result.
