  An oft-ignored challenge of real-world reinforcement learning is that the
real world does not pause when agents make learning updates. As standard
simulated environments do not address this real-time aspect of learning, most
available implementations of RL algorithms process environment interactions and
learning updates sequentially. As a consequence, when such implementations are
deployed in the real world, they may make decisions based on significantly
delayed observations and not act responsively. Asynchronous learning has been
proposed to solve this issue, but no systematic comparison between sequential
and asynchronous reinforcement learning was conducted using real-world
environments. In this work, we set up two vision-based tasks with a robotic
arm, implement an asynchronous learning system that extends a previous
architecture, and compare sequential and asynchronous reinforcement learning
across different action cycle times, sensory data dimensions, and mini-batch
sizes. Our experiments show that when the time cost of learning updates
increases, the action cycle time in sequential implementation could grow
excessively long, while the asynchronous implementation can always maintain an
appropriate action cycle time. Consequently, when learning updates are
expensive, the performance of sequential learning diminishes and is
outperformed by asynchronous learning by a substantial margin. Our system
learns in real-time to reach and track visual targets from pixels within two
hours of experience and does so directly using real robots, learning completely
from scratch.
