  The success of reinforcement learning for real world robotics has been, in
many cases limited to instrumented laboratory scenarios, often requiring
arduous human effort and oversight to enable continuous learning. In this work,
we discuss the elements that are needed for a robotic learning system that can
continually and autonomously improve with data collected in the real world. We
propose a particular instantiation of such a system, using dexterous
manipulation as our case study. Subsequently, we investigate a number of
challenges that come up when learning without instrumentation. In such
settings, learning must be feasible without manually designed resets, using
only on-board perception, and without hand-engineered reward functions. We
propose simple and scalable solutions to these challenges, and then demonstrate
the efficacy of our proposed system on a set of dexterous robotic manipulation
tasks, providing an in-depth analysis of the challenges associated with this
learning paradigm. We demonstrate that our complete system can learn without
any human intervention, acquiring a variety of vision-based skills with a
real-world three-fingered hand. Results and videos can be found at
https://sites.google.com/view/realworld-rl/
