  This paper focuses on a novel and challenging vision task, dense video
captioning, which aims to automatically describe a video clip with multiple
informative and diverse caption sentences. The proposed method is trained
without explicit annotation of fine-grained sentence to video region-sequence
correspondence, but is only based on weak video-level sentence annotations. It
differs from existing video captioning systems in three technical aspects.
First, we propose lexical fully convolutional neural networks (Lexical-FCN)
with weakly supervised multi-instance multi-label learning to weakly link video
regions with lexical labels. Second, we introduce a novel submodular
maximization scheme to generate multiple informative and diverse
region-sequences based on the Lexical-FCN outputs. A winner-takes-all scheme is
adopted to weakly associate sentences to region-sequences in the training
phase. Third, a sequence-to-sequence learning based language model is trained
with the weakly supervised information obtained through the association
process. We show that the proposed method can not only produce informative and
diverse dense captions, but also outperform state-of-the-art single video
captioning methods by a large margin.
