  Temporal sentence grounding (TSG) aims to localize the temporal segment which
is semantically aligned with a natural language query in an untrimmed
video.Most existing methods extract frame-grained features or object-grained
features by 3D ConvNet or detection network under a conventional TSG framework,
failing to capture the subtle differences between frames or to model the
spatio-temporal behavior of core persons/objects. In this paper, we introduce a
new perspective to address the TSG task by tracking pivotal objects and
activities to learn more fine-grained spatio-temporal behaviors. Specifically,
we propose a novel Temporal Sentence Tracking Network (TSTNet), which contains
(A) a Cross-modal Targets Generator to generate multi-modal templates and
search space, filtering objects and activities, and (B) a Temporal Sentence
Tracker to track multi-modal targets for modeling the targets' behavior and to
predict query-related segment. Extensive experiments and comparisons with
state-of-the-arts are conducted on challenging benchmarks: Charades-STA and
TACoS. And our TSTNet achieves the leading performance with a considerable
real-time speed.
