  Predicting the future in real-world settings, particularly from raw sensory
observations such as images, is exceptionally challenging. Real-world events
can be stochastic and unpredictable, and the high dimensionality and complexity
of natural images requires the predictive model to build an intricate
understanding of the natural world. Many existing methods tackle this problem
by making simplifying assumptions about the environment. One common assumption
is that the outcome is deterministic and there is only one plausible future.
This can lead to low-quality predictions in real-world settings with stochastic
dynamics. In this paper, we develop a stochastic variational video prediction
(SV2P) method that predicts a different possible future for each sample of its
latent variables. To the best of our knowledge, our model is the first to
provide effective stochastic multi-frame prediction for real-world video. We
demonstrate the capability of the proposed method in predicting detailed future
frames of videos on multiple real-world datasets, both action-free and
action-conditioned. We find that our proposed method produces substantially
improved video predictions when compared to the same model without
stochasticity, and to other stochastic video prediction methods. Our SV2P
implementation will be open sourced upon publication.
