  We present a novel data-driven simulation environment for modeling traffic in
metropolitan street intersections. Using real-world tracking data collected
over an extended period of time, we train trajectory forecasting models to
learn agent interactions and environmental constraints that are difficult to
capture conventionally. Trajectories of new agents are first coarsely generated
by sampling from the spatial and temporal generative distributions, then
refined using state-of-the-art trajectory forecasting models. The simulation
can run either autonomously, or under explicit human control conditioned on the
generative distributions. We present the experiments for a variety of model
configurations. Under an iterative prediction scheme, the way-point-supervised
TrajNet++ model obtained 0.36 Final Displacement Error (FDE) in 20 FPS on an
NVIDIA A100 GPU.
