  This paper tackles video prediction from a new dimension of predicting
spacetime-varying motions that are incessantly changing across both space and
time. Prior methods mainly capture the temporal state transitions but overlook
the complex spatiotemporal variations of the motion itself, making them
difficult to adapt to ever-changing motions. We observe that physical world
motions can be decomposed into transient variation and motion trend, while the
latter can be regarded as the accumulation of previous motions. Thus,
simultaneously capturing the transient variation and the motion trend is the
key to make spacetime-varying motions more predictable. Based on these
observations, we propose the MotionRNN framework, which can capture the complex
variations within motions and adapt to spacetime-varying scenarios. MotionRNN
has two main contributions. The first is that we design the MotionGRU unit,
which can model the transient variation and motion trend in a unified way. The
second is that we apply the MotionGRU to RNN-based predictive models and
indicate a new flexible video prediction architecture with a Motion Highway
that can significantly improve the ability to predict changeable motions and
avoid motion vanishing for stacked multiple-layer predictive models. With high
flexibility, this framework can adapt to a series of models for deterministic
spatiotemporal prediction. Our MotionRNN can yield significant improvements on
three challenging benchmarks for video prediction with spacetime-varying
motions.
