  The goal of weakly-supervised video moment retrieval is to localize the video
segment most relevant to the given natural language query without access to
temporal annotations during training. Prior strongly- and weakly-supervised
approaches often leverage co-attention mechanisms to learn visual-semantic
representations for localization. However, while such approaches tend to focus
on identifying relationships between elements of the video and language
modalities, there is less emphasis on modeling relational context between video
frames given the semantic context of the query. Consequently, the
above-mentioned visual-semantic representations, built upon local frame
features, do not contain much contextual information. To address this
limitation, we propose a Latent Graph Co-Attention Network (LoGAN) that
exploits fine-grained frame-by-word interactions to reason about
correspondences between all possible pairs of frames, given the semantic
context of the query. Comprehensive experiments across two datasets, DiDeMo and
Charades-Sta, demonstrate the effectiveness of our proposed latent co-attention
model where it outperforms current state-of-the-art (SOTA) weakly-supervised
approaches by a significant margin. Notably, it even achieves a 11% improvement
to Recall@1 accuracy over strongly-supervised SOTA methods on DiDeMo.
