  We present a benchmark to facilitate simulated manipulation; an attempt to
overcome the obstacles of physical benchmarks through the distribution of a
real world, ground truth dataset. Users are given various simulated
manipulation tasks with assigned protocols having the objective of replicating
the real world results of a recorded dataset. The benchmark comprises of a
range of metrics used to characterise the successes of submitted environments
whilst providing insight into their deficiencies. We apply our benchmark to two
simulation environments, PyBullet and V-Rep, and publish the results. All
materials required to benchmark an environment, including protocols and the
dataset, can be found at the benchmarks' website
https://research.csiro.au/robotics/manipulation-benchmark/.
