  Existing studies on multimodal sentiment analysis heavily rely on textual
modality and unavoidably induce the spurious correlations between textual words
and sentiment labels. This greatly hinders the model generalization ability. To
address this problem, we define the task of out-of-distribution (OOD)
multimodal sentiment analysis. This task aims to estimate and mitigate the bad
effect of textual modality for strong OOD generalization. To this end, we
embrace causal inference, which inspects the causal relationships via a causal
graph. From the graph, we find that the spurious correlations are attributed to
the direct effect of textual modality on the model prediction while the
indirect one is more reliable by considering multimodal semantics. Inspired by
this, we devise a model-agnostic counterfactual framework for multimodal
sentiment analysis, which captures the direct effect of textual modality via an
extra text model and estimates the indirect one by a multimodal model. During
the inference, we first estimate the direct effect by the counterfactual
inference, and then subtract it from the total effect of all modalities to
obtain the indirect effect for reliable prediction. Extensive experiments show
the superior effectiveness and generalization ability of our proposed
framework.
