  Models that can simulate how environments change in response to actions can
be used by agents to plan and act efficiently. We improve on previous
environment simulators from high-dimensional pixel observations by introducing
recurrent neural networks that are able to make temporally and spatially
coherent predictions for hundreds of time-steps into the future. We present an
in-depth analysis of the factors affecting performance, providing the most
extensive attempt to advance the understanding of the properties of these
models. We address the issue of computationally inefficiency with a model that
does not need to generate a high-dimensional image at each time-step. We show
that our approach can be used to improve exploration and is adaptable to many
diverse environments, namely 10 Atari games, a 3D car racing environment, and
complex 3D mazes.
