  We present a self-supervised learning approach to learn audio-visual
representations from video and audio. Our method uses contrastive learning for
cross-modal discrimination of video from audio and vice-versa. We show that
optimizing for cross-modal discrimination, rather than within-modal
discrimination, is important to learn good representations from video and
audio. With this simple but powerful insight, our method achieves highly
competitive performance when finetuned on action recognition tasks.
Furthermore, while recent work in contrastive learning defines positive and
negative samples as individual instances, we generalize this definition by
exploring cross-modal agreement. We group together multiple instances as
positives by measuring their similarity in both the video and audio feature
spaces. Cross-modal agreement creates better positive and negative sets, which
allows us to calibrate visual similarities by seeking within-modal
discrimination of positive instances, and achieve significant gains on
downstream tasks.
