  Video activity localisation has recently attained increasing attention due to
its practical values in automatically localising the most salient visual
segments corresponding to their language descriptions (sentences) from
untrimmed and unstructured videos. For supervised model training, a temporal
annotation of both the start and end time index of each video segment for a
sentence (a video moment) must be given. This is not only very expensive but
also sensitive to ambiguity and subjective annotation bias, a much harder task
than image labelling. In this work, we develop a more accurate
weakly-supervised solution by introducing Cross-Sentence Relations Mining (CRM)
in video moment proposal generation and matching when only a paragraph
description of activities without per-sentence temporal annotation is
available. Specifically, we explore two cross-sentence relational constraints:
(1) Temporal ordering and (2) semantic consistency among sentences in a
paragraph description of video activities. Existing weakly-supervised
techniques only consider within-sentence video segment correlations in training
without considering cross-sentence paragraph context. This can mislead due to
ambiguous expressions of individual sentences with visually indiscriminate
video moment proposals in isolation. Experiments on two publicly available
activity localisation datasets show the advantages of our approach over the
state-of-the-art weakly supervised methods, especially so when the video
activity descriptions become more complex.
