  A realistic simulation environment is an essential tool in every roboticist's
toolkit, with uses ranging from planning and control to training policies with
reinforcement learning. Despite the centrality of simulation in modern
robotics, little work has been done to compare the performance of robotics
simulators against real-world data, especially for scenarios involving dynamic
motions with high speed impact events. Handling dynamic contact is the
computational bottleneck for most simulations, and thus the modeling and
algorithmic choices surrounding impacts and friction form the largest
distinctions between popular tools. Here, we evaluate the ability of several
simulators to reproduce real-world trajectories involving impacts. Using
experimental data, we identify system-specific contact parameters of popular
simulators Drake, MuJoCo, and Bullet, analyzing the effects of modeling choices
around these parameters. For the simple example of a cube tossed onto a table,
simulators capture inelastic impacts well while failing to capture elastic
impacts. For the higher-dimensional case of a Cassie biped landing from a jump,
the simulators capture the bulk motion well but the accuracy is limited by
numerous model differences between the real robot and the simulators.
