  Video summarization aims to create short, accurate, and cohesive summaries of
longer videos. Despite the existence of various video summarization datasets, a
notable limitation is their limited amount of source videos, which hampers the
effective fine-tuning of advanced large vision-language models (VLMs).
Additionally, most existing datasets are created for video-to-video
summarization, overlooking the contemporary need for multimodal video content
summarization. Recent efforts have been made to expand from unimodal to
multimodal video summarization, categorizing the task into three sub-tasks
based on the summary's modality: video-to-video (V2V), video-to-text (V2T), and
a combination of video and text summarization (V2VT). However, the textual
summaries in previous multimodal datasets are inadequate. To address these
issues, we introduce Instruct-V2Xum, a cross-modal video summarization dataset
featuring 30,000 diverse videos sourced from YouTube, with lengths ranging from
40 to 940 seconds and an average summarization ratio of 16.39\%. Each video
summary in Instruct-V2Xum is paired with a textual summary that references
specific frame indexes, facilitating the generation of aligned video and
textual summaries. In addition, we propose a new video summarization framework
named V2Xum-LLM. V2Xum-LLM, specifically V2Xum-LLaMA in this study, is the
first framework that unifies different video summarization tasks into one large
language model's (LLM) text decoder and achieves task-controllable video
summarization with temporal prompts and task instructions. Experiments show
that V2Xum-LLaMA outperforms strong baseline models on multiple video
summarization tasks. Furthermore, we propose an enhanced evaluation metric for
V2V and V2VT summarization tasks.
