  We study the problem of localizing audio-visual events that are both audible
and visible in a video. Existing works focus on encoding and aligning audio and
visual features at the segment level while neglecting informative correlation
between segments of the two modalities and between multi-scale event proposals.
We propose a novel MultiModulation Network (M2N) to learn the above correlation
and leverage it as semantic guidance to modulate the related auditory, visual,
and fused features. In particular, during feature encoding, we propose
cross-modal normalization and intra-modal normalization. The former modulates
the features of two modalities by establishing and exploiting the cross-modal
relationship. The latter modulates the features of a single modality with the
event-relevant semantic guidance of the same modality. In the fusion stage,we
propose a multi-scale proposal modulating module and a multi-alignment segment
modulating module to introduce multi-scale event proposals and enable dense
matching between cross-modal segments. With the auditory, visual, and fused
features modulated by the correlation information regarding audio-visual
events, M2N performs accurate event localization. Extensive experiments
conducted on the AVE dataset demonstrate that our proposed method outperforms
the state of the art in both supervised event localization and cross-modality
localization.
