  Integration of multimodal information from various sources has been shown to
boost the performance of machine learning models and thus has received
increased attention in recent years. Often such models use deep
modality-specific networks to obtain unimodal features which are combined to
obtain "late-fusion" representations. However, these designs run the risk of
information loss in the respective unimodal pipelines. On the other hand,
"early-fusion" methodologies, which combine features early, suffer from the
problems associated with feature heterogeneity and high sample complexity. In
this work, we present an iterative representation refinement approach, called
Progressive Fusion, which mitigates the issues with late fusion
representations. Our model-agnostic technique introduces backward connections
that make late stage fused representations available to early layers, improving
the expressiveness of the representations at those stages, while retaining the
advantages of late fusion designs. We test Progressive Fusion on tasks
including affective sentiment detection, multimedia analysis, and time series
fusion with different models, demonstrating its versatility. We show that our
approach consistently improves performance, for instance attaining a 5%
reduction in MSE and 40% improvement in robustness on multimodal time series
prediction.
