  The recent success of audio-visual representation learning can be largely
attributed to their pervasive property of audio-visual synchronization, which
can be used as self-annotated supervision. As a state-of-the-art solution,
Audio-Visual Instance Discrimination (AVID) extends instance discrimination to
the audio-visual realm. Existing AVID methods construct the contrastive set by
random sampling based on the assumption that the audio and visual clips from
all other videos are not semantically related. We argue that this assumption is
rough, since the resulting contrastive sets have a large number of faulty
negatives. In this paper, we overcome this limitation by proposing a novel
Active Contrastive Set Mining (ACSM) that aims to mine the contrastive sets
with informative and diverse negatives for robust AVID. Moreover, we also
integrate a semantically-aware hard-sample mining strategy into our ACSM. The
proposed ACSM is implemented into two most recent state-of-the-art AVID methods
and significantly improves their performance. Extensive experiments conducted
on both action and sound recognition on multiple datasets show the remarkably
improved performance of our method.
