  We address the challenging task of cross-modal moment retrieval, which aims
to localize a temporal segment from an untrimmed video described by a natural
language query. It poses great challenges over the proper semantic alignment
between vision and linguistic domains. Existing methods independently extract
the features of videos and sentences and purely utilize the sentence embedding
in the multi-modal fusion stage, which do not make full use of the potential of
language. In this paper, we present Language Guided Networks (LGN), a new
framework that leverages the sentence embedding to guide the whole process of
moment retrieval. In the first feature extraction stage, we propose to jointly
learn visual and language features to capture the powerful visual information
which can cover the complex semantics in the sentence query. Specifically, the
early modulation unit is designed to modulate the visual feature extractor's
feature maps by a linguistic embedding. Then we adopt a multi-modal fusion
module in the second fusion stage. Finally, to get a precise localizer, the
sentence information is utilized to guide the process of predicting temporal
positions. Specifically, the late guidance module is developed to linearly
transform the output of localization networks via the channel attention
mechanism. The experimental results on two popular datasets demonstrate the
superior performance of our proposed method on moment retrieval (improving by
5.8\% in terms of<EMAIL_ADDRESS>on Charades-STA and 5.2\% on TACoS). The source
code for the complete system will be publicly available.
