  Standardized evaluation measures have aided in the progress of machine
learning approaches in disciplines such as computer vision and machine
translation. In this paper, we make the case that robotic learning would also
benefit from benchmarking, and present the "REPLAB" platform for benchmarking
vision-based manipulation tasks. REPLAB is a reproducible and self-contained
hardware stack (robot arm, camera, and workspace) that costs about 2000 USD,
occupies a cuboid of size 70x40x60 cm, and permits full assembly within a few
hours. Through this low-cost, compact design, REPLAB aims to drive wide
participation by lowering the barrier to entry into robotics and to enable easy
scaling to many robots. We envision REPLAB as a framework for reproducible
research across manipulation tasks, and as a step in this direction, we define
a template for a grasping benchmark consisting of a task definition, evaluation
protocol, performance measures, and a dataset of 92k grasp attempts. We
implement, evaluate, and analyze several previously proposed grasping
approaches to establish baselines for this benchmark. Finally, we also
implement and evaluate a deep reinforcement learning approach for 3D reaching
tasks on our REPLAB platform. Project page with assembly instructions, code,
and videos: https://goo.gl/5F9dP4.
