  Deep reinforcement learning has emerged as a popular and powerful way to
develop locomotion controllers for quadruped robots. Common approaches have
largely focused on learning actions directly in joint space, or learning to
modify and offset foot positions produced by trajectory generators. Both
approaches typically require careful reward shaping and training for millions
of time steps, and with trajectory generators introduce human bias into the
resulting control policies. In this paper, we present a learning framework that
leads to the natural emergence of fast and robust bounding policies for
quadruped robots. The agent both selects and controls actions directly in task
space to track desired velocity commands subject to environmental noise
including model uncertainty and rough terrain. We observe that this framework
improves sample efficiency, necessitates little reward shaping, leads to the
emergence of natural gaits such as galloping and bounding, and eases the
sim-to-real transfer at running speeds. Policies can be learned in only a few
million time steps, even for challenging tasks of running over rough terrain
with loads of over 100% of the nominal quadruped mass. Training occurs in
PyBullet, and we perform a sim-to-sim transfer to Gazebo and sim-to-real
transfer to the Unitree A1 hardware. For sim-to-sim, our results show the
quadruped is able to run at over 4 m/s without a load, and 3.5 m/s with a 10 kg
load, which is over 83% of the nominal quadruped mass. For sim-to-real, the
Unitree A1 is able to bound at 2 m/s with a 5 kg load, representing 42% of the
nominal quadruped mass.
