  Deep reinforcement learning (RL) can enable robots to autonomously acquire
complex behaviors, such as legged locomotion. However, RL in the real world is
complicated by constraints on efficiency, safety, and overall training
stability, which limits its practical applicability. We present APRL, a policy
regularization framework that modulates the robot's exploration over the course
of training, striking a balance between flexible improvement potential and
focused, efficient exploration. APRL enables a quadrupedal robot to efficiently
learn to walk entirely in the real world within minutes and continue to improve
with more training where prior work saturates in performance. We demonstrate
that continued training with APRL results in a policy that is substantially
more capable of navigating challenging situations and is able to adapt to
changes in dynamics with continued training.
