  Multimodal headline utilizes both video frames and transcripts to generate
the natural language title of the videos. Due to a lack of large-scale,
manually annotated data, the task of annotating grounded headlines for video is
labor intensive and impractical. Previous researches on pre-trained language
models and video-language models have achieved significant progress in related
downstream tasks. However, none of them can be directly applied to multimodal
headline architecture where we need both multimodal encoder and sentence
decoder. A major challenge in simply gluing language model and video-language
model is the modality balance, which is aimed at combining visual-language
complementary abilities. In this paper, we propose a novel approach to graft
the video encoder from the pre-trained video-language model on the generative
pre-trained language model. We also present a consensus fusion mechanism for
the integration of different components, via inter/intra modality relation.
Empirically, experiments show that the grafted model achieves strong results on
a brand-new dataset collected from real-world applications.
