Learning Compositional Neural Programs for Continuous Control
  We propose a novel solution to challenging sparse-reward, continuous control
problems that require hierarchical planning at multiple levels of abstraction.
Our solution, dubbed AlphaNPI-X, involves three separate stages of learning.
First, we use off-policy reinforcement learning algorithms with experience
replay to learn a set of atomic goal-conditioned policies, which can be easily
repurposed for many tasks. Second, we learn self-models describing the effect
of the atomic policies on the environment. Third, the self-models are harnessed
to learn recursive compositional programs with multiple levels of abstraction.
The key insight is that the self-models enable planning by imagination,
obviating the need for interaction with the world when learning higher-level
compositional programs. To accomplish the third stage of learning, we extend
the AlphaNPI algorithm, which applies AlphaZero to learn recursive neural
programmer-interpreters. We empirically show that AlphaNPI-X can effectively
learn to tackle challenging sparse manipulation tasks, such as stacking
multiple blocks, where powerful model-free baselines fail.
