  In this paper we undertake the task of text-based video moment retrieval from
a corpus of videos. To train the model, text-moment paired datasets were used
to learn the correct correspondences. In typical training methods, ground-truth
text-moment pairs are used as positive pairs, whereas other pairs are regarded
as negative pairs. However, aside from the ground-truth pairs, some text-moment
pairs should be regarded as positive. In this case, one text annotation can be
positive for many video moments. Conversely, one video moment can be
corresponded to many text annotations. Thus, there are many-to-many
correspondences between the text annotations and video moments. Based on these
correspondences, we can form potentially relevant pairs, which are not given as
ground truth yet are not negative; effectively incorporating such relevant
pairs into training can improve the retrieval performance. The text query
should describe what is happening in a video moment. Hence, different video
moments annotated with similar texts, which contain a similar action, are
likely to hold the similar action, thus these pairs can be considered as
potentially relevant pairs. In this paper, we propose a novel training method
that takes advantage of potentially relevant pairs, which are detected based on
linguistic analysis about text annotation. Experiments on two benchmark
datasets revealed that our method improves the retrieval performance both
quantitatively and qualitatively.
