  Previous works on scene classification are mainly based on audio or visual
signals, while humans perceive the environmental scenes through multiple
senses. Recent studies on audio-visual scene classification separately
fine-tune the largescale audio and image pre-trained models on the target
dataset, then either fuse the intermediate representations of the audio model
and the visual model, or fuse the coarse-grained decision of both models at the
clip level. Such methods ignore the detailed audio events and visual objects in
audio-visual scenes (AVS), while humans often identify different scenes through
audio events and visual objects within and the congruence between them. To
exploit the fine-grained information of audio events and visual objects in AVS,
and coordinate the implicit relationship between audio events and visual
objects, this paper proposes a multibranch model equipped with contrastive
event-object alignment (CEOA) and semantic-based fusion (SF) for AVSC. CEOA
aims to align the learned embeddings of audio events and visual objects by
comparing the difference between audio-visual event-object pairs. Then, visual
objects associated with certain audio events and vice versa are accentuated by
cross-attention and undergo SF for semantic-level fusion. Experiments show
that: 1) the proposed AVSC model equipped with CEOA and SF outperforms the
results of audio-only and visual-only models, i.e., the audio-visual results
are better than the results from a single modality. 2) CEOA aligns the
embeddings of audio events and related visual objects on a fine-grained level,
and the SF effectively integrates both; 3) Compared with other large-scale
integrated systems, the proposed model shows competitive performance, even
without using additional datasets and data augmentation tricks.
