  Detecting customized moments and highlights from videos given natural
language (NL) user queries is an important but under-studied topic. One of the
challenges in pursuing this direction is the lack of annotated data. To address
this issue, we present the Query-based Video Highlights (QVHIGHLIGHTS) dataset.
It consists of over 10,000 YouTube videos, covering a wide range of topics,
from everyday activities and travel in lifestyle vlog videos to social and
political activities in news videos. Each video in the dataset is annotated
with: (1) a human-written free-form NL query, (2) relevant moments in the video
w.r.t. the query, and (3) five-point scale saliency scores for all
query-relevant clips. This comprehensive annotation enables us to develop and
evaluate systems that detect relevant moments as well as salient highlights for
diverse, flexible user queries. We also present a strong baseline for this
task, Moment-DETR, a transformer encoder-decoder model that views moment
retrieval as a direct set prediction problem, taking extracted video and query
representations as inputs and predicting moment coordinates and saliency scores
end-to-end. While our model does not utilize any human prior, we show that it
performs competitively when compared to well-engineered architectures. With
weakly supervised pretraining using ASR captions, MomentDETR substantially
outperforms previous methods. Lastly, we present several ablations and
visualizations of Moment-DETR. Data and code is publicly available at
https://github.com/jayleicn/moment_detr
