  Most existing video-and-language (VidL) research focuses on a single dataset,
or multiple datasets of a single task. In reality, a truly useful VidL system
is expected to be easily generalizable to diverse tasks, domains, and datasets.
To facilitate the evaluation of such systems, we introduce Video-And-Language
Understanding Evaluation (VALUE) benchmark, an assemblage of 11 VidL datasets
over 3 popular tasks: (i) text-to-video retrieval; (ii) video question
answering; and (iii) video captioning. VALUE benchmark aims to cover a broad
range of video genres, video lengths, data volumes, and task difficulty levels.
Rather than focusing on single-channel videos with visual information only,
VALUE promotes models that leverage information from both video frames and
their associated subtitles, as well as models that share knowledge across
multiple tasks. We evaluate various baseline methods with and without
large-scale VidL pre-training, and systematically investigate the impact of
video input channels, fusion methods, and different video representations. We
also study the transferability between tasks, and conduct multi-task learning
under different settings. The significant gap between our best model and human
performance calls for future study for advanced VidL models. VALUE is available
at https://value-benchmark.github.io/.
