  Information integration from different modalities is an active area of
research. Human beings and, in general, biological neural systems are quite
adept at using a multitude of signals from different sensory perceptive fields
to interact with the environment and each other. Recent work in deep fusion
models via neural networks has led to substantial improvements over unimodal
approaches in areas like speech recognition, emotion recognition and analysis,
captioning and image description. However, such research has mostly focused on
architectural changes allowing for fusion of different modalities while keeping
the model complexity manageable. Inspired by recent neuroscience ideas about
multisensory integration and processing, we investigate the effect of synergy
maximizing loss functions. Experiments on multimodal sentiment analysis tasks:
CMU-MOSI and CMU-MOSEI with different models show that our approach provides a
consistent performance boost.
