  Recently, audio-visual scene classification (AVSC) has attracted increasing
attention from multidisciplinary communities. Previous studies tended to adopt
a pipeline training strategy, which uses well-trained visual and acoustic
encoders to extract high-level representations (embeddings) first, then
utilizes them to train the audio-visual classifier. In this way, the extracted
embeddings are well suited for uni-modal classifiers, but not necessarily
suited for multi-modal ones. In this paper, we propose a joint training
framework, using the acoustic features and raw images directly as inputs for
the AVSC task. Specifically, we retrieve the bottom layers of pre-trained image
models as visual encoder, and jointly optimize the scene classifier and 1D-CNN
based acoustic encoder during training. We evaluate the approach on the
development dataset of TAU Urban Audio-Visual Scenes 2021. The experimental
results show that our proposed approach achieves significant improvement over
the conventional pipeline training strategy. Moreover, our best single system
outperforms previous state-of-the-art methods, yielding a log loss of 0.1517
and accuracy of 94.59% on the official test fold.
