  While traditional methods for instruction-following typically assume prior
linguistic and perceptual knowledge, many recent works in reinforcement
learning (RL) have proposed learning policies end-to-end, typically by training
neural networks to map joint representations of observations and instructions
directly to actions. In this work, we present a novel framework for learning to
perform temporally extended tasks using spatial reasoning in the RL framework,
by sequentially imagining visual goals and choosing appropriate actions to
fulfill imagined goals. Our framework operates on raw pixel images, assumes no
prior linguistic or perceptual knowledge, and learns via intrinsic motivation
and a single extrinsic reward signal measuring task completion. We validate our
method in two environments with a robot arm in a simulated interactive 3D
environment. Our method outperforms two flat architectures with raw-pixel and
ground-truth states, and a hierarchical architecture with ground-truth states
on object arrangement tasks.
