  The core of video understanding tasks, such as recognition, captioning, and
tracking, is to automatically detect objects or actions in a video and analyze
their temporal evolution. Despite sharing a common goal, different tasks often
rely on distinct model architectures and annotation formats. In contrast,
natural language processing benefits from a unified output space, i.e., text
sequences, which simplifies the training of powerful foundational language
models, such as GPT-3, with extensive training corpora. Inspired by this, we
seek to unify the output space of video understanding tasks by using languages
as labels and additionally introducing time and box tokens. In this way, a
variety of video tasks could be formulated as video-grounded token generation.
This enables us to address various types of video tasks, including
classification (such as action recognition), captioning (covering clip
captioning, video question answering, and dense video captioning), and
localization tasks (such as visual object tracking) within a fully shared
encoder-decoder architecture, following a generative framework. Through
comprehensive experiments, we demonstrate such a simple and straightforward
idea is quite effective and can achieve state-of-the-art or competitive results
on seven video benchmarks, providing a novel perspective for more universal
video understanding. Code is available at https://github.com/wangjk666/OmniVid.
