  In recent years, researchers combine both audio and video signals to deal
with challenges where actions are not well represented or captured by visual
cues. However, how to effectively leverage the two modalities is still under
development. In this work, we develop a multiscale multimodal Transformer (MMT)
that leverages hierarchical representation learning. Particularly, MMT is
composed of a novel multiscale audio Transformer (MAT) and a multiscale video
Transformer [43]. To learn a discriminative cross-modality fusion, we further
design multimodal supervised contrastive objectives called audio-video
contrastive loss (AVC) and intra-modal contrastive loss (IMC) that robustly
align the two modalities. MMT surpasses previous state-of-the-art approaches by
7.3% and 2.1% on Kinetics-Sounds and VGGSound in terms of the top-1 accuracy
without external training data. Moreover, the proposed MAT significantly
outperforms AST [28] by 22.2%, 4.4% and 4.7% on three public benchmark
datasets, and is about 3% more efficient based on the number of FLOPs and 9.8%
more efficient based on GPU memory usage.
