  Interactive multi-agent simulation algorithms are used to compute the
trajectories and behaviors of different entities in virtual reality scenarios.
However, current methods involve considerable parameter tweaking to generate
plausible behaviors. We introduce a novel approach (Heter-Sim) that combines
physics-based simulation methods with data-driven techniques using an
optimization-based formulation. Our approach is general and can simulate
heterogeneous agents corresponding to human crowds, traffic, vehicles, or
combinations of different agents with varying dynamics. We estimate motion
states from real-world datasets that include information about position,
velocity, and control direction. Our optimization algorithm considers several
constraints, including velocity continuity, collision avoidance, attraction,
and direction control. To accelerate the computations, we reduce the search
space for both collision avoidance and optimal solution computation. Heter-Sim
can simulate tens or hundreds of agents at interactive rates and we compare its
accuracy with real-world datasets and prior algorithms. We also perform user
studies that evaluate the plausible behaviors generated by our algorithm and a
user study that evaluates the plausibility of our algorithm via VR.
