  An intelligent observer looks at the world and sees not only what is, but
what is moving and what can be moved. In other words, the observer sees how the
present state of the world can transform in the future. We propose a model that
predicts future images by learning to represent the present state and its
transformation given only a sequence of images. To do so, we introduce an
architecture with a latent state composed of two components designed to capture
(i) the present image state and (ii) the transformation between present and
future states, respectively. We couple this latent state with a recurrent
neural network (RNN) core that predicts future frames by transforming past
states into future states by applying the accumulated state transformation with
a learned operator. We describe how this model can be integrated into an
encoder-decoder convolutional neural network (CNN) architecture that uses
weighted residual connections to integrate representations of the past with
representations of the future. Qualitatively, our approach generates image
sequences that are stable and capture realistic motion over multiple predicted
frames, without requiring adversarial training. Quantitatively, our method
achieves prediction results comparable to state-of-the-art results on standard
image prediction benchmarks (Moving MNIST, KTH, and UCF101).
