  Smart traffic lights in intelligent transportation systems (ITSs) are
envisioned to greatly increase traffic efficiency and reduce congestion. Deep
reinforcement learning (DRL) is a promising approach to adaptively control
traffic lights based on the real-time traffic situation in a road network.
However, conventional methods may suffer from poor scalability. In this paper,
we investigate deep reinforcement learning to control traffic lights, and both
theoretical analysis and numerical experiments show that the intelligent
behavior ``greenwave" (i.e., a vehicle will see a progressive cascade of green
lights, and not have to brake at any intersection) emerges naturally a grid
road network, which is proved to be the optimal policy in an avenue with
multiple cross streets. As a first step, we use two DRL algorithms for the
traffic light control problems in two scenarios. In a single road intersection,
we verify that the deep Q-network (DQN) algorithm delivers a thresholding
policy; and in a grid road network, we adopt the deep deterministic policy
gradient (DDPG) algorithm. Secondly, numerical experiments show that the DQN
algorithm delivers the optimal control, and the DDPG algorithm with passive
observations has the capability to produce on its own a high-level intelligent
behavior in a grid road network, namely, the ``greenwave" policy emerges. We
also verify the ``greenwave" patterns in a $5 \times 10$ grid road network.
Thirdly, the ``greenwave" patterns demonstrate that DRL algorithms produce
favorable solutions since the ``greenwave" policy shown in experiment results
is proved to be optimal in a specified traffic model (an avenue with multiple
cross streets). The delivered policies both in a single road intersection and a
grid road network demonstrate the scalability of DRL algorithms.
