  Temporal grounding aims to localize a video moment which is semantically
aligned with a given natural language query. Existing methods typically apply a
detection or regression pipeline on the fused representation with the research
focus on designing complicated prediction heads or fusion strategies. Instead,
from a perspective on temporal grounding as a metric-learning problem, we
present a Mutual Matching Network (MMN), to directly model the similarity
between language queries and video moments in a joint embedding space. This new
metric-learning framework enables fully exploiting negative samples from two
new aspects: constructing negative cross-modal pairs in a mutual matching
scheme and mining negative pairs across different videos. These new negative
samples could enhance the joint representation learning of two modalities via
cross-modal mutual matching to maximize their mutual information. Experiments
show that our MMN achieves highly competitive performance compared with the
state-of-the-art methods on four video grounding benchmarks. Based on MMN, we
present a winner solution for the HC-STVG challenge of the 3rd PIC workshop.
This suggests that metric learning is still a promising method for temporal
grounding via capturing the essential cross-modal correlation in a joint
embedding space. Code is available at https://github.com/MCG-NJU/MMN.
