  We address the problem of retrieving a specific moment from an untrimmed
video by a query sentence. This is a challenging problem because a target
moment may take place in relations to other temporal moments in the untrimmed
video. Existing methods cannot tackle this challenge well since they consider
temporal moments individually and neglect the temporal dependencies. In this
paper, we model the temporal relations between video moments by a
two-dimensional map, where one dimension indicates the starting time of a
moment and the other indicates the end time. This 2D temporal map can cover
diverse video moments with different lengths, while representing their adjacent
relations. Based on the 2D map, we propose a Temporal Adjacent Network
(2D-TAN), a single-shot framework for moment localization. It is capable of
encoding the adjacent temporal relation, while learning discriminative features
for matching video moments with referring expressions. We evaluate the proposed
2D-TAN on three challenging benchmarks, i.e., Charades-STA, ActivityNet
Captions, and TACoS, where our 2D-TAN outperforms the state-of-the-art.
