  Highlight detection has the potential to significantly ease video browsing,
but existing methods often suffer from expensive supervision requirements,
where human viewers must manually identify highlights in training videos. We
propose a scalable unsupervised solution that exploits video duration as an
implicit supervision signal. Our key insight is that video segments from
shorter user-generated videos are more likely to be highlights than those from
longer videos, since users tend to be more selective about the content when
capturing shorter videos. Leveraging this insight, we introduce a novel ranking
framework that prefers segments from shorter videos, while properly accounting
for the inherent noise in the (unlabeled) training data. We use it to train a
highlight detector with 10M hashtagged Instagram videos. In experiments on two
challenging public video highlight detection benchmarks, our method
substantially improves the state-of-the-art for unsupervised highlight
detection.
