  Generating controllable videos conforming to user intentions is an appealing
yet challenging topic in computer vision. To enable maneuverable control in
line with user intentions, a novel video generation task, named
Text-Image-to-Video generation (TI2V), is proposed. With both controllable
appearance and motion, TI2V aims at generating videos from a static image and a
text description. The key challenges of TI2V task lie both in aligning
appearance and motion from different modalities, and in handling uncertainty in
text descriptions. To address these challenges, we propose a Motion
Anchor-based video GEnerator (MAGE) with an innovative motion anchor (MA)
structure to store appearance-motion aligned representation. To model the
uncertainty and increase the diversity, it further allows the injection of
explicit condition and implicit randomness. Through three-dimensional axial
transformers, MA is interacted with given image to generate next frames
recursively with satisfying controllability and diversity. Accompanying the new
task, we build two new video-text paired datasets based on MNIST and CATER for
evaluation. Experiments conducted on these datasets verify the effectiveness of
MAGE and show appealing potentials of TI2V task. Source code for model and
datasets will be available soon.
