  Realistic traffic simulation is crucial for developing self-driving software
in a safe and scalable manner prior to real-world deployment. Typically,
imitation learning (IL) is used to learn human-like traffic agents directly
from real-world observations collected offline, but without explicit
specification of traffic rules, agents trained from IL alone frequently display
unrealistic infractions like collisions and driving off the road. This problem
is exacerbated in out-of-distribution and long-tail scenarios. On the other
hand, reinforcement learning (RL) can train traffic agents to avoid
infractions, but using RL alone results in unhuman-like driving behaviors. We
propose Reinforcing Traffic Rules (RTR), a holistic closed-loop learning
objective to match expert demonstrations under a traffic compliance constraint,
which naturally gives rise to a joint IL + RL approach, obtaining the best of
both worlds. Our method learns in closed-loop simulations of both nominal
scenarios from real-world datasets as well as procedurally generated long-tail
scenarios. Our experiments show that RTR learns more realistic and
generalizable traffic simulation policies, achieving significantly better
tradeoffs between human-like driving and traffic compliance in both nominal and
long-tail scenarios. Moreover, when used as a data generation tool for training
prediction models, our learned traffic policy leads to considerably improved
downstream prediction metrics compared to baseline traffic agents. For more
information, visit the project website: https://waabi.ai/rtr
