  Generating video stories from text prompts is a complex task. In addition to
having high visual quality, videos need to realistically adhere to a sequence
of text prompts whilst being consistent throughout the frames. Creating a
benchmark for video generation requires data annotated over time, which
contrasts with the single caption used often in video datasets. To fill this
gap, we collect comprehensive human annotations on three existing datasets, and
introduce StoryBench: a new, challenging multi-task benchmark to reliably
evaluate forthcoming text-to-video models. Our benchmark includes three video
generation tasks of increasing difficulty: action execution, where the next
action must be generated starting from a conditioning video; story
continuation, where a sequence of actions must be executed starting from a
conditioning video; and story generation, where a video must be generated from
only text prompts. We evaluate small yet strong text-to-video baselines, and
show the benefits of training on story-like data algorithmically generated from
existing video captions. Finally, we establish guidelines for human evaluation
of video stories, and reaffirm the need of better automatic metrics for video
generation. StoryBench aims at encouraging future research efforts in this
exciting new area.
