  Video highlight detection is a crucial yet challenging problem that aims to
identify the interesting moments in untrimmed videos. The key to this task lies
in effective video representations that jointly pursue two goals,
\textit{i.e.}, cross-modal representation learning and fine-grained feature
discrimination. In this paper, these two challenges are tackled by not only
enriching intra-modality and cross-modality relations for representation
modeling but also shaping the features in a discriminative manner. Our proposed
method mainly leverages the intra-modality encoding and cross-modality
co-occurrence encoding for fully representation modeling. Specifically,
intra-modality encoding augments the modality-wise features and dampens
irrelevant modality via within-modality relation learning in both audio and
visual signals. Meanwhile, cross-modality co-occurrence encoding focuses on the
co-occurrence inter-modality relations and selectively captures effective
information among multi-modality. The multi-modal representation is further
enhanced by the global information abstracted from the local context. In
addition, we enlarge the discriminative power of feature embedding with a
hard-pairs guided contrastive learning (HPCL) scheme. A hard-pairs sampling
strategy is further employed to mine the hard samples for improving feature
discrimination in HPCL. Extensive experiments conducted on two benchmarks
demonstrate the effectiveness and superiority of our proposed methods compared
to other state-of-the-art methods.
