  Dense video captioning, a task of localizing meaningful moments and
generating relevant captions for videos, often requires a large, expensive
corpus of annotated video segments paired with text. In an effort to minimize
the annotation cost, we propose ZeroTA, a novel method for dense video
captioning in a zero-shot manner. Our method does not require any videos or
annotations for training; instead, it localizes and describes events within
each input video at test time by optimizing solely on the input. This is
accomplished by introducing a soft moment mask that represents a temporal
segment in the video and jointly optimizing it with the prefix parameters of a
language model. This joint optimization aligns a frozen language generation
model (i.e., GPT-2) with a frozen vision-language contrastive model (i.e.,
CLIP) by maximizing the matching score between the generated text and a moment
within the video. We also introduce a pairwise temporal IoU loss to let a set
of soft moment masks capture multiple distinct events within the video. Our
method effectively discovers diverse significant events within the video, with
the resulting captions appropriately describing these events. The empirical
results demonstrate that ZeroTA surpasses zero-shot baselines and even
outperforms the state-of-the-art few-shot method on the widely-used benchmark
ActivityNet Captions. Moreover, our method shows greater robustness compared to
supervised methods when evaluated in out-of-domain scenarios. This research
provides insight into the potential of aligning widely-used models, such as
language generation models and vision-language models, to unlock a new
capability: understanding temporal aspects of videos.
