  We introduce the task of retrieving relevant video moments from a large
corpus of untrimmed, unsegmented videos given a natural language query. Our
task poses unique challenges as a system must efficiently identify both the
relevant videos and localize the relevant moments in the videos. To address
these challenges, we propose SpatioTemporal Alignment with Language (STAL), a
model that represents a video moment as a set of regions within a series of
short video clips and aligns a natural language query to the moment's regions.
Our alignment cost compares variable-length language and video features using
symmetric squared Chamfer distance, which allows for efficient indexing and
retrieval of the video moments. Moreover, aligning language features to regions
within a video moment allows for finer alignment compared to methods that
extract only an aggregate feature from the entire video moment. We evaluate our
approach on two recently proposed datasets for temporal localization of moments
in video with natural language (DiDeMo and Charades-STA) extended to our video
corpus moment retrieval setting. We show that our STAL re-ranking model
outperforms the recently proposed Moment Context Network on all criteria across
all datasets on our proposed task, obtaining relative gains of 37% - 118% for
average recall and up to 30% for median rank. Moreover, our approach achieves
more than 130x faster retrieval and 8x smaller index size with a 1M video
corpus in an approximate setting.
