  We present CrissCross, a self-supervised framework for learning audio-visual
representations. A novel notion is introduced in our framework whereby in
addition to learning the intra-modal and standard 'synchronous' cross-modal
relations, CrissCross also learns 'asynchronous' cross-modal relationships. We
perform in-depth studies showing that by relaxing the temporal synchronicity
between the audio and visual modalities, the network learns strong generalized
representations useful for a variety of downstream tasks. To pretrain our
proposed solution, we use 3 different datasets with varying sizes,
Kinetics-Sound, Kinetics400, and AudioSet. The learned representations are
evaluated on a number of downstream tasks namely action recognition, sound
classification, and action retrieval. Our experiments show that CrissCross
either outperforms or achieves performances on par with the current
state-of-the-art self-supervised methods on action recognition and action
retrieval with UCF101 and HMDB51, as well as sound classification with ESC50
and DCASE. Moreover, CrissCross outperforms fully-supervised pretraining while
pretrained on Kinetics-Sound. The codes and pretrained models are available on
the project website.
