  Video-Language Pre-training models have recently significantly improved
various multi-modal downstream tasks. Previous dominant works mainly adopt
contrastive learning to achieve global feature alignment across modalities.
However, the local associations between videos and texts are not modeled,
restricting the pre-training models' generality, especially for tasks requiring
the temporal video boundary for certain query texts. This work introduces a
novel text-video localization pre-text task to enable fine-grained temporal and
semantic alignment such that the trained model can accurately perceive temporal
boundaries in videos given the text description. Specifically, text-video
localization consists of moment retrieval, which predicts start and end
boundaries in videos given the text description, and text localization which
matches the subset of texts with the video features. To produce temporal
boundaries, frame features in several videos are manually merged into a long
video sequence that interacts with a text sequence. With the localization task,
our method connects the fine-grained frame representations with the word
representations and implicitly distinguishes representations of different
instances in the single modality. Notably, comprehensive experimental results
show that our method significantly improves the state-of-the-art performance on
various benchmarks, covering text-to-video retrieval, video question answering,
video captioning, temporal action localization and temporal moment retrieval.
The code will be released soon.
