  In this letter, we formulate a novel Markov Decision Process (MDP) for safe
and data-efficient learning for humanoid locomotion aided by a dynamic
balancing model. In our previous studies of biped locomotion, we relied on a
low-dimensional robot model, commonly used in high-level Walking Pattern
Generators (WPGs). However, a low-level feedback controller cannot precisely
track desired footstep locations due to the discrepancies between the full
order model and the simplified model. In this study, we propose mitigating this
problem by complementing a WPG with reinforcement learning. More specifically,
we propose a structured footstep control method consisting of a WPG, a neural
network, and a safety controller. The WPG provides an analytical method that
promotes efficient learning while the neural network maximizes long-term
rewards, and the safety controller encourages safe exploration based on step
capturability and the use of control-barrier functions. Our contributions
include the following (1) a structured learning control method for locomotion,
(2) a data-efficient and safe learning process to improve walking using a
physics-based model, and (3) the scalability of the procedure to various types
of humanoid robots and walking.
