  As audio/visual classification models are widely deployed for sensitive tasks
like content filtering at scale, it is critical to understand their robustness
along with improving the accuracy. This work aims to study several key
questions related to multimodal learning through the lens of adversarial
noises: 1) The trade-off between early/middle/late fusion affecting its
robustness and accuracy 2) How do different frequency/time domain features
contribute to the robustness? 3) How do different neural modules contribute to
the adversarial noise? In our experiment, we construct adversarial examples to
attack state-of-the-art neural models trained on Google AudioSet. We compare
how much attack potency in terms of adversarial perturbation of size $\epsilon$
using different $L_p$ norms we would need to "deactivate" the victim model.
Using adversarial noise to ablate multimodal models, we are able to provide
insights into what is the best potential fusion strategy to balance the model
parameters/accuracy and robustness trade-off and distinguish the robust
features versus the non-robust features that various neural networks model tend
to learn.
