  In this paper, we consider a novel task, Spatio-Temporal Video Grounding for
Multi-Form Sentences (STVG). Given an untrimmed video and a
declarative/interrogative sentence depicting an object, STVG aims to localize
the spatio-temporal tube of the queried object. STVG has two challenging
settings: (1) We need to localize spatio-temporal object tubes from untrimmed
videos, where the object may only exist in a very small segment of the video;
(2) We deal with multi-form sentences, including the declarative sentences with
explicit objects and interrogative sentences with unknown objects. Existing
methods cannot tackle the STVG task due to the ineffective tube pre-generation
and the lack of object relationship modeling. Thus, we then propose a novel
Spatio-Temporal Graph Reasoning Network (STGRN) for this task. First, we build
a spatio-temporal region graph to capture the region relationships with
temporal object dynamics, which involves the implicit and explicit spatial
subgraphs in each frame and the temporal dynamic subgraph across frames. We
then incorporate textual clues into the graph and develop the multi-step
cross-modal graph reasoning. Next, we introduce a spatio-temporal localizer
with a dynamic selection method to directly retrieve the spatio-temporal tubes
without tube pre-generation. Moreover, we contribute a large-scale video
grounding dataset VidSTG based on video relation dataset VidOR. The extensive
experiments demonstrate the effectiveness of our method.
