  We propose Video Localized Narratives, a new form of multimodal video
annotations connecting vision and language. In the original Localized
Narratives, annotators speak and move their mouse simultaneously on an image,
thus grounding each word with a mouse trace segment. However, this is
challenging on a video. Our new protocol empowers annotators to tell the story
of a video with Localized Narratives, capturing even complex events involving
multiple actors interacting with each other and with several passive objects.
We annotated 20k videos of the OVIS, UVO, and Oops datasets, totalling 1.7M
words. Based on this data, we also construct new benchmarks for the video
narrative grounding and video question answering tasks, and provide reference
results from strong baseline models. Our annotations are available at
https://google.github.io/video-localized-narratives/.
