  Current video generation models usually convert signals indicating appearance
and motion received from inputs (e.g., image, text) or latent spaces (e.g.,
noise vectors) into consecutive frames, fulfilling a stochastic generation
process for the uncertainty introduced by latent code sampling. However, this
generation pattern lacks deterministic constraints for both appearance and
motion, leading to uncontrollable and undesirable outcomes. To this end, we
propose a new task called Text-driven Video Prediction (TVP). Taking the first
frame and text caption as inputs, this task aims to synthesize the following
frames. Specifically, appearance and motion components are provided by the
image and caption separately. The key to addressing the TVP task depends on
fully exploring the underlying motion information in text descriptions, thus
facilitating plausible video generation. In fact, this task is intrinsically a
cause-and-effect problem, as the text content directly influences the motion
changes of frames. To investigate the capability of text in causal inference
for progressive motion information, our TVP framework contains a Text Inference
Module (TIM), producing step-wise embeddings to regulate motion inference for
subsequent frames. In particular, a refinement mechanism incorporating global
motion semantics guarantees coherent generation. Extensive experiments are
conducted on Something-Something V2 and Single Moving MNIST datasets.
Experimental results demonstrate that our model achieves better results over
other baselines, verifying the effectiveness of the proposed framework.
