  The performance of video prediction has been greatly boosted by advanced deep
neural networks. However, most of the current methods suffer from large model
sizes and require extra inputs, e.g., semantic/depth maps, for promising
performance. For efficiency consideration, in this paper, we propose a Dynamic
Multi-scale Voxel Flow Network (DMVFN) to achieve better video prediction
performance at lower computational costs with only RGB images, than previous
methods. The core of our DMVFN is a differentiable routing module that can
effectively perceive the motion scales of video frames. Once trained, our DMVFN
selects adaptive sub-networks for different inputs at the inference stage.
Experiments on several benchmarks demonstrate that our DMVFN is an order of
magnitude faster than Deep Voxel Flow and surpasses the state-of-the-art
iterative-based OPT on generated image quality. Our code and demo are available
at https://huxiaotaostasy.github.io/DMVFN/.
