  Generating a video given the first several static frames is challenging as it
anticipates reasonable future frames with temporal coherence. Besides video
prediction, the ability to rewind from the last frame or infilling between the
head and tail is also crucial, but they have rarely been explored for video
completion. Since there could be different outcomes from the hints of just a
few frames, a system that can follow natural language to perform video
completion may significantly improve controllability. Inspired by this, we
introduce a novel task, text-guided video completion (TVC), which requests the
model to generate a video from partial frames guided by an instruction. We then
propose Multimodal Masked Video Generation (MMVG) to address this TVC task.
During training, MMVG discretizes the video frames into visual tokens and masks
most of them to perform video completion from any time point. At inference
time, a single MMVG model can address all 3 cases of TVC, including video
prediction, rewind, and infilling, by applying corresponding masking
conditions. We evaluate MMVG in various video scenarios, including egocentric,
animation, and gaming. Extensive experimental results indicate that MMVG is
effective in generating high-quality visual appearances with text guidance for
TVC.
