  Multiple modalities can provide more valuable information than single one by
describing the same contents in various ways. Hence, it is highly expected to
learn effective joint representation by fusing the features of different
modalities. However, previous methods mainly focus on fusing the shallow
features or high-level representations generated by unimodal deep networks,
which only capture part of the hierarchical correlations across modalities. In
this paper, we propose to densely integrate the representations by greedily
stacking multiple shared layers between different modality-specific networks,
which is named as Dense Multimodal Fusion (DMF). The joint representations in
different shared layers can capture the correlations in different levels, and
the connection between shared layers also provides an efficient way to learn
the dependence among hierarchical correlations. These two properties jointly
contribute to the multiple learning paths in DMF, which results in faster
convergence, lower training loss, and better performance. We evaluate our model
on three typical multimodal learning tasks, including audiovisual speech
recognition, cross-modal retrieval, and multimodal classification. The
noticeable performance in the experiments demonstrates that our model can learn
more effective joint representation.
