  Through many recent successes in simulation, model-free reinforcement
learning has emerged as a promising approach to solving continuous control
robotic tasks. The research community is now able to reproduce, analyze and
build quickly on these results due to open source implementations of learning
algorithms and simulated benchmark tasks. To carry forward these successes to
real-world applications, it is crucial to withhold utilizing the unique
advantages of simulations that do not transfer to the real world and experiment
directly with physical robots. However, reinforcement learning research with
physical robots faces substantial resistance due to the lack of benchmark tasks
and supporting source code. In this work, we introduce several reinforcement
learning tasks with multiple commercially available robots that present varying
levels of learning difficulty, setup, and repeatability. On these tasks, we
test the learning performance of off-the-shelf implementations of four
reinforcement learning algorithms and analyze sensitivity to their
hyper-parameters to determine their readiness for applications in various
real-world tasks. Our results show that with a careful setup of the task
interface and computations, some of these implementations can be readily
applicable to physical robots. We find that state-of-the-art learning
algorithms are highly sensitive to their hyper-parameters and their relative
ordering does not transfer across tasks, indicating the necessity of re-tuning
them for each task for best performance. On the other hand, the best
hyper-parameter configuration from one task may often result in effective
learning on held-out tasks even with different robots, providing a reasonable
default. We make the benchmark tasks publicly available to enhance
reproducibility in real-world reinforcement learning.
