  The Audio Visual Question Answering (AVQA) task aims to answer questions
related to various visual objects, sounds, and their interactions in videos.
Such naturally multimodal videos contain rich and complex dynamic audio-visual
components, with only a portion of them closely related to the given questions.
Hence, effectively perceiving audio-visual cues relevant to the given questions
is crucial for correctly answering them. In this paper, we propose a
Temporal-Spatial Perception Model (TSPM), which aims to empower the model to
perceive key visual and auditory cues related to the questions. Specifically,
considering the challenge of aligning non-declarative questions and visual
representations into the same semantic space using visual-language pretrained
models, we construct declarative sentence prompts derived from the question
template, to assist the temporal perception module in better identifying
critical segments relevant to the questions. Subsequently, a spatial perception
module is designed to merge visual tokens from selected segments to highlight
key latent targets, followed by cross-modal interaction with audio to perceive
potential sound-aware areas. Finally, the significant temporal-spatial cues
from these modules are integrated to answer the question. Extensive experiments
on multiple AVQA benchmarks demonstrate that our framework excels not only in
understanding audio-visual scenes but also in answering complex questions
effectively. Code is available at https://github.com/GeWu-Lab/TSPM.
