  System optimal traffic routing can mitigate congestion by assigning routes
for a portion of vehicles so that the total travel time of all vehicles in the
transportation system can be reduced. However, achieving real-time optimal
routing poses challenges due to uncertain demands and unknown system dynamics,
particularly in expansive transportation networks. While physics model-based
methods are sensitive to uncertainties and model mismatches, model-free
reinforcement learning struggles with learning inefficiencies and
interpretability issues. Our paper presents TransRL, a novel algorithm that
integrates reinforcement learning with physics models for enhanced performance,
reliability, and interpretability. TransRL begins by establishing a
deterministic policy grounded in physics models, from which it learns from and
is guided by a differentiable and stochastic teacher policy. During training,
TransRL aims to maximize cumulative rewards while minimizing the Kullback
Leibler (KL) divergence between the current policy and the teacher policy. This
approach enables TransRL to simultaneously leverage interactions with the
environment and insights from physics models. We conduct experiments on three
transportation networks with up to hundreds of links. The results demonstrate
TransRL's superiority over traffic model-based methods for being adaptive and
learning from the actual network data. By leveraging the information from
physics models, TransRL consistently outperforms state-of-the-art reinforcement
learning algorithms such as proximal policy optimization (PPO) and soft actor
critic (SAC). Moreover, TransRL's actions exhibit higher reliability and
interpretability compared to baseline reinforcement learning approaches like
PPO and SAC.
