  Video moment retrieval (MR) and highlight detection (HD) based on natural
language queries are two highly related tasks, which aim to obtain relevant
moments within videos and highlight scores of each video clip. Recently,
several methods have been devoted to building DETR-based networks to solve both
MR and HD jointly. These methods simply add two separate task heads after
multi-modal feature extraction and feature interaction, achieving good
performance. Nevertheless, these approaches underutilize the reciprocal
relationship between two tasks. In this paper, we propose a task-reciprocal
transformer based on DETR (TR-DETR) that focuses on exploring the inherent
reciprocity between MR and HD. Specifically, a local-global multi-modal
alignment module is first built to align features from diverse modalities into
a shared latent space. Subsequently, a visual feature refinement is designed to
eliminate query-irrelevant information from visual features for modal
interaction. Finally, a task cooperation module is constructed to refine the
retrieval pipeline and the highlight score prediction process by utilizing the
reciprocity between MR and HD. Comprehensive experiments on QVHighlights,
Charades-STA and TVSum datasets demonstrate that TR-DETR outperforms existing
state-of-the-art methods. Codes are available at
\url{https://github.com/mingyao1120/TR-DETR}.
