  Audio-Visual Video Parsing (AVVP) task aims to detect and temporally locate
events within audio and visual modalities. Multiple events can overlap in the
timeline, making identification challenging. While traditional methods usually
focus on improving the early audio-visual encoders to embed more effective
features, the decoding phase -- crucial for final event classification, often
receives less attention. We aim to advance the decoding phase and improve its
interpretability. Specifically, we introduce a new decoding paradigm,
\underline{l}abel s\underline{e}m\underline{a}ntic-based \underline{p}rojection
(LEAP), that employs labels texts of event categories, each bearing distinct
and explicit semantics, for parsing potentially overlapping events.LEAP works
by iteratively projecting encoded latent features of audio/visual segments onto
semantically independent label embeddings. This process, enriched by modeling
cross-modal (audio/visual-label) interactions, gradually disentangles event
semantics within video segments to refine relevant label embeddings,
guaranteeing a more discriminative and interpretable decoding process. To
facilitate the LEAP paradigm, we propose a semantic-aware optimization
strategy, which includes a novel audio-visual semantic similarity loss
function. This function leverages the Intersection over Union of audio and
visual events (EIoU) as a novel metric to calibrate audio-visual similarities
at the feature level, accommodating the varied event densities across
modalities. Extensive experiments demonstrate the superiority of our method,
achieving new state-of-the-art performance for AVVP and also enhancing the
relevant audio-visual event localization task.
