  Multimodal learning has exhibited a significant advantage in affective
analysis tasks owing to the comprehensive information of various modalities,
particularly the complementary information. Thus, many emerging studies focus
on disentangling the modality-invariant and modality-specific representations
from input data and then fusing them for prediction. However, our study shows
that modality-specific representations may contain information that is
irrelevant or conflicting with the tasks, which downgrades the effectiveness of
learned multimodal representations. We revisit the disentanglement issue, and
propose a novel triple disentanglement approach, TriDiRA, which disentangles
the modality-invariant, effective modality-specific and ineffective
modality-specific representations from input data. By fusing only the
modality-invariant and effective modality-specific representations, TriDiRA can
significantly alleviate the impact of irrelevant and conflicting information
across modalities during model training. Extensive experiments conducted on
four benchmark datasets demonstrate the effectiveness and generalization of our
triple disentanglement, which outperforms SOTA methods.
