  Temporal sentence localization in videos (TSLV) aims to retrieve the most
interested segment in an untrimmed video according to a given sentence query.
However, almost of existing TSLV approaches suffer from the same limitations:
(1) They only focus on either frame-level or object-level visual representation
learning and corresponding correlation reasoning, but fail to integrate them
both; (2) They neglect to leverage the rich semantic contexts to further
benefit the query reasoning. To address these issues, in this paper, we propose
a novel Hierarchical Visual- and Semantic-Aware Reasoning Network (HVSARN),
which enables both visual- and semantic-aware query reasoning from object-level
to frame-level. Specifically, we present a new graph memory mechanism to
perform visual-semantic query reasoning: For visual reasoning, we design a
visual graph memory to leverage visual information of video; For semantic
reasoning, a semantic graph memory is also introduced to explicitly leverage
semantic knowledge contained in the classes and attributes of video objects,
and perform correlation reasoning in the semantic space. Experiments on three
datasets demonstrate that our HVSARN achieves a new state-of-the-art
performance.
