  The use of multi-modal data for deep machine learning has shown promise when
compared to uni-modal approaches with fusion of multi-modal features resulting
in improved performance in several applications. However, most state-of-the-art
methods use naive fusion which processes feature streams independently,
ignoring possible long-term dependencies within the data during fusion. In this
paper, we present a novel Memory based Attentive Fusion layer, which fuses
modes by incorporating both the current features and longterm dependencies in
the data, thus allowing the model to understand the relative importance of
modes over time. We introduce an explicit memory block within the fusion layer
which stores features containing long-term dependencies of the fused data. The
feature inputs from uni-modal encoders are fused through attentive composition
and transformation followed by naive fusion of the resultant memory derived
features with layer inputs. Following state-of-the-art methods, we have
evaluated the performance and the generalizability of the proposed fusion
approach on two different datasets with different modalities. In our
experiments, we replace the naive fusion layer in benchmark networks with our
proposed layer to enable a fair comparison. Experimental results indicate that
the MBAF layer can generalise across different modalities and networks to
enhance fusion and improve performance.
