  In light of the challenges and costs of real-world testing, autonomous
vehicle developers often rely on testing in simulation for the creation of
reliable systems. A key element of effective simulation is the incorporation of
realistic traffic models that align with human knowledge, an aspect that has
proven challenging due to the need to balance realism and diversity. This works
aims to address this by developing a framework that employs reinforcement
learning with human preference (RLHF) to enhance the realism of existing
traffic models. This study also identifies two main challenges: capturing the
nuances of human preferences on realism and the unification of diverse traffic
simulation models. To tackle these issues, we propose using human feedback for
alignment and employ RLHF due to its sample efficiency. We also introduce the
first dataset for realism alignment in traffic modeling to support such
research. Our framework, named TrafficRLHF, demonstrates its proficiency in
generating realistic traffic scenarios that are well-aligned with human
preferences, as corroborated by comprehensive evaluations on the nuScenes
dataset.
