  In Self-Supervised Learning (SSL), Audio-Visual Correspondence (AVC) is a
popular task to learn deep audio and video features from large unlabeled
datasets. The key step in AVC is to randomly sample audio and video clips from
the dataset and learn to minimize the feature distance between the positive
pairs (corresponding audio-video pair) while maximizing the distance between
the negative pairs (non-corresponding audio-video pairs). The learnt features
are shown to be effective on various downstream tasks. However, these methods
achieve subpar performance when the size of the dataset is rather small. In
this paper, we investigate the effect of utilizing class label information in
the AVC feature learning task. We modified various positive and negative data
sampling techniques of SSL based on class label information to investigate the
effect on the feature quality. We propose a new sampling approach which we call
soft-positive sampling, where the positive pair for one audio sample is not
from the exact corresponding video, but from a video of the same class.
Experimental results suggest that when the dataset size is small in SSL setup,
features learnt through the soft-positive sampling method significantly
outperform those from the traditional SSL sampling approaches. This trend holds
in both in-domain and out-of-domain downstream tasks, and even outperforms
supervised classification. Finally, experiments show that class label
information can easily be obtained using a publicly available classifier
network and then can be used to boost the SSL performance without adding extra
data annotation burden.
