  Humans can abstract prior knowledge from very little data and use it to boost
skill learning. In this paper, we propose routine-augmented policy learning
(RAPL), which discovers routines composed of primitive actions from a single
demonstration and uses discovered routines to augment policy learning. To
discover routines from the demonstration, we first abstract routine candidates
by identifying grammar over the demonstrated action trajectory. Then, the best
routines measured by length and frequency are selected to form a routine
library. We propose to learn policy simultaneously at primitive-level and
routine-level with discovered routines, leveraging the temporal structure of
routines. Our approach enables imitating expert behavior at multiple temporal
scales for imitation learning and promotes reinforcement learning exploration.
Extensive experiments on Atari games demonstrate that RAPL improves the
state-of-the-art imitation learning method SQIL and reinforcement learning
method A2C. Further, we show that discovered routines can generalize to unseen
levels and difficulties on the CoinRun benchmark.
