  This study proposes a delay-compensated feedback controller based on proximal
policy optimization (PPO) reinforcement learning to stabilize traffic flow in
the congested regime by manipulating the time-gap of adaptive cruise
control-equipped (ACC-equipped) vehicles.The traffic dynamics on a freeway
segment are governed by an Aw-Rascle-Zhang (ARZ) model, consisting of $2\times
2$ nonlinear first-order partial differential equations (PDEs).Inspired by the
backstepping delay compensator [18] but different from whose complex segmented
control scheme, the PPO control is composed of three feedbacks, namely the
current traffic flow velocity, the current traffic flow density and previous
one step control input. The control gains for the three feedbacks are learned
from the interaction between the PPO and the numerical simulator of the traffic
system without knowing the system dynamics. Numerical simulation experiments
are designed to compare the Lyapunov control, the backstepping control and the
PPO control. The results show that for a delay-free system, the PPO control has
faster convergence rate and less control effort than the Lyapunov control. For
a traffic system with input delay, the performance of the PPO controller is
comparable to that of the Backstepping controller, even for the situation that
the delay value does not match. However, the PPO is robust to parameter
perturbations, while the Backstepping controller cannot stabilize a system
where one of the parameters is disturbed by Gaussian noise.
