  Three challenges limit the progress of robot learning research: robots are
expensive (few labs can participate), everyone uses different robots (findings
do not generalize across labs), and we lack internet-scale robotics data. We
take on these challenges via a new benchmark: Train Offline, Test Online
(TOTO). TOTO provides remote users with access to shared robotic hardware for
evaluating methods on common tasks and an open-source dataset of these tasks
for offline training. Its manipulation task suite requires challenging
generalization to unseen objects, positions, and lighting. We present initial
results on TOTO comparing five pretrained visual representations and four
offline policy learning baselines, remotely contributed by five institutions.
The real promise of TOTO, however, lies in the future: we release the benchmark
for additional submissions from any user, enabling easy, direct comparison to
several methods without the need to obtain hardware or collect data.
