  With the growth of high-quality data and advancement in visual pre-training
paradigms, Video Foundation Models (VFMs) have made significant progress
recently, demonstrating their remarkable performance on traditional video
understanding benchmarks. However, the existing benchmarks (e.g. Kinetics) and
their evaluation protocols are often limited by relatively poor diversity, high
evaluation costs, and saturated performance metrics. In this paper, we build a
comprehensive benchmark suite to address these issues, namely VideoEval.
Specifically, we establish the Video Task Adaption Benchmark (VidTAB) and the
Video Embedding Benchmark (VidEB) from two perspectives: evaluating the task
adaptability of VFMs under few-shot conditions and assessing their
representation power by directly applying to downstream tasks. With VideoEval,
we conduct a large-scale study on 20 popular open-source vision foundation
models. Our study reveals some insightful findings on VFMs: 1) overall, current
VFMs exhibit weak generalization across diverse tasks, 2) increasing video
data, whether labeled or weakly-labeled video-text pairs, does not necessarily
improve task performance, 3) the effectiveness of some pre-training paradigms
may not be fully validated in previous benchmarks, and 4) combining different
pre-training paradigms can help improve the generalization capabilities. We
believe this study serves as an important complement to the current evaluation
for VFMs and offers valuable insights for the future research.
