  Motion is an important cue for video prediction and often utilized by
separating video content into static and dynamic components. Most of the
previous work utilizing motion is deterministic but there are stochastic
methods that can model the inherent uncertainty of the future. Existing
stochastic models either do not reason about motion explicitly or make limiting
assumptions about the static part. In this paper, we reason about appearance
and motion in the video stochastically by predicting the future based on the
motion history. Explicit reasoning about motion without history already reaches
the performance of current stochastic models. The motion history further
improves the results by allowing to predict consistent dynamics several frames
into the future. Our model performs comparably to the state-of-the-art models
on the generic video prediction datasets, however, significantly outperforms
them on two challenging real-world autonomous driving datasets with complex
motion and dynamic background.
