  With the exponential growth of video content, the need for automated video
highlight detection to extract key moments or highlights from lengthy videos
has become increasingly pressing. This technology has the potential to
significantly enhance user experiences by allowing quick access to relevant
content across diverse domains. Existing methods typically rely either on
expensive manually labeled frame-level annotations, or on a large external
dataset of videos for weak supervision through category information. To
overcome this, we focus on unsupervised video highlight detection, eliminating
the need for manual annotations. We propose an innovative unsupervised approach
which capitalizes on the premise that significant moments tend to recur across
multiple videos of the similar category in both audio and visual modalities.
Surprisingly, audio remains under-explored, especially in unsupervised
algorithms, despite its potential to detect key moments. Through a clustering
technique, we identify pseudo-categories of videos and compute audio
pseudo-highlight scores for each video by measuring the similarities of audio
features among audio clips of all the videos within each pseudo-category.
Similarly, we also compute visual pseudo-highlight scores for each video using
visual features. Subsequently, we combine audio and visual pseudo-highlights to
create the audio-visual pseudo ground-truth highlight of each video for
training an audio-visual highlight detection network. Extensive experiments and
ablation studies on three highlight detection benchmarks showcase the superior
performance of our method over prior work.
