  While great strides have been made in using deep learning algorithms to solve
supervised learning tasks, the problem of unsupervised learning - leveraging
unlabeled examples to learn about the structure of a domain - remains a
difficult unsolved challenge. Here, we explore prediction of future frames in a
video sequence as an unsupervised learning rule for learning about the
structure of the visual world. We describe a predictive neural network
("PredNet") architecture that is inspired by the concept of "predictive coding"
from the neuroscience literature. These networks learn to predict future frames
in a video sequence, with each layer in the network making local predictions
and only forwarding deviations from those predictions to subsequent network
layers. We show that these networks are able to robustly learn to predict the
movement of synthetic (rendered) objects, and that in doing so, the networks
learn internal representations that are useful for decoding latent object
parameters (e.g. pose) that support object recognition with fewer training
views. We also show that these networks can scale to complex natural image
streams (car-mounted camera videos), capturing key aspects of both egocentric
movement and the movement of objects in the visual scene, and the
representation learned in this setting is useful for estimating the steering
angle. Altogether, these results suggest that prediction represents a powerful
framework for unsupervised learning, allowing for implicit learning of object
and scene structure.
