  Cross-modal correlation provides an inherent supervision for video
unsupervised representation learning. Existing methods focus on distinguishing
different video clips by visual and audio representations. We human visual
perception could attend to regions where sounds are made, and our auditory
perception could also ground their frequencies of sounding objects, which we
call bidirectional local correspondence. Such supervision is intuitive but not
well explored in the contrastive learning framework. This paper introduces a
pretext task, Cross-Modal Attention Consistency (CMAC), for exploring the
bidirectional local correspondence property. The CMAC approach aims to align
the regional attention generated purely from the visual signal with the target
attention generated under the guidance of acoustic signal, and do a similar
alignment for frequency grounding on the acoustic attention. Accompanied by a
remoulded cross-modal contrastive loss where we consider additional
within-modal interactions, the CMAC approach works effectively for enforcing
the bidirectional alignment. Extensive experiments on six downstream benchmarks
demonstrate that CMAC can improve the state-of-the-art performance on both
visual and audio modalities.
