  In this paper, we describe an approach to achieve dynamic legged locomotion
on physical robots which combines existing methods for control with
reinforcement learning. Specifically, our goal is a control hierarchy in which
highest-level behaviors are planned through reduced-order models, which
describe the fundamental physics of legged locomotion, and lower level
controllers utilize a learned policy that can bridge the gap between the
idealized, simple model and the complex, full order robot. The high-level
planner can use a model of the environment and be task specific, while the
low-level learned controller can execute a wide range of motions so that it
applies to many different tasks. In this letter we describe this learned
dynamic walking controller and show that a range of walking motions from
reduced-order models can be used as the command and primary training signal for
learned policies. The resulting policies do not attempt to naively track the
motion (as a traditional trajectory tracking controller would) but instead
balance immediate motion tracking with long term stability. The resulting
controller is demonstrated on a human scale, unconstrained, untethered bipedal
robot at speeds up to 1.2 m/s. This letter builds the foundation of a generic,
dynamic learned walking controller that can be applied to many different tasks.
