  Learning generic joint representations for video and text by a supervised
method requires a prohibitively substantial amount of manually annotated video
datasets. As a practical alternative, a large-scale but uncurated and narrated
video dataset, HowTo100M, has recently been introduced. But it is still
challenging to learn joint embeddings of video and text in a self-supervised
manner, due to its ambiguity and non-sequential alignment. In this paper, we
propose a novel multi-modal self-supervised framework Video-Text Temporally
Weak Alignment-based Contrastive Learning (VT-TWINS) to capture significant
information from noisy and weakly correlated data using a variant of Dynamic
Time Warping (DTW). We observe that the standard DTW inherently cannot handle
weakly correlated data and only considers the globally optimal alignment path.
To address these problems, we develop a differentiable DTW which also reflects
local information with weak temporal alignment. Moreover, our proposed model
applies a contrastive learning scheme to learn feature representations on
weakly correlated data. Our extensive experiments demonstrate that VT-TWINS
attains significant improvements in multi-modal representation learning and
outperforms various challenging downstream tasks. Code is available at
https://github.com/mlvlab/VT-TWINS.
