  Weakly supervised audio-visual video parsing (AVVP) methods aim to detect
audible-only, visible-only, and audible-visible events using only video-level
labels. Existing approaches tackle this by leveraging unimodal and cross-modal
contexts. However, we argue that while cross-modal learning is beneficial for
detecting audible-visible events, in the weakly supervised scenario, it
negatively impacts unaligned audible or visible events by introducing
irrelevant modality information. In this paper, we propose CoLeaF, a novel
learning framework that optimizes the integration of cross-modal context in the
embedding space such that the network explicitly learns to combine cross-modal
information for audible-visible events while filtering them out for unaligned
events. Additionally, as videos often involve complex class relationships,
modelling them improves performance. However, this introduces extra
computational costs into the network. Our framework is designed to leverage
cross-class relationships during training without incurring additional
computations at inference. Furthermore, we propose new metrics to better
evaluate a method's capabilities in performing AVVP. Our extensive experiments
demonstrate that CoLeaF significantly improves the state-of-the-art results by
an average of 1.9% and 2.4% F-score on the LLP and UnAV-100 datasets,
respectively.
