  Simulators are a critical component of modern robotics research. Strategies
for both perception and decision making can be studied in simulation first
before deployed to real world systems, saving on time and costs. Despite
significant progress on the development of sim-to-real algorithms, the analysis
of different methods is still conducted in an ad-hoc manner, without a
consistent set of tests and metrics for comparison. This paper fills this gap
and proposes a set of benchmarks and a framework for the study of various
algorithms aimed to transfer models and policies learnt in simulation to the
real world. We conduct experiments on a wide range of well known simulated
environments to characterize and offer insights into the performance of
different algorithms. Our analysis can be useful for practitioners working in
this area and can help make informed choices about the behavior and main
properties of sim-to-real algorithms. We open-source the benchmark, training
data, and trained models, which can be found at
https://github.com/NVlabs/sim-parameter-estimation.
