  This paper proposes Video-Teller, a video-language foundation model that
leverages multi-modal fusion and fine-grained modality alignment to
significantly enhance the video-to-text generation task. Video-Teller boosts
the training efficiency by utilizing frozen pretrained vision and language
modules. It capitalizes on the robust linguistic capabilities of large language
models, enabling the generation of both concise and elaborate video
descriptions. To effectively integrate visual and auditory information,
Video-Teller builds upon the image-based BLIP-2 model and introduces a cascaded
Q-Former which fuses information across frames and ASR texts. To better guide
video summarization, we introduce a fine-grained modality alignment objective,
where the cascaded Q-Former's output embedding is trained to align with the
caption/summary embedding created by a pretrained text auto-encoder.
Experimental results demonstrate the efficacy of our proposed video-language
foundation model in accurately comprehending videos and generating coherent and
precise language descriptions. It is worth noting that the fine-grained
alignment enhances the model's capabilities (4% improvement of CIDEr score on
MSR-VTT) with only 13% extra parameters in training and zero additional cost in
inference.
