Abstractive Sentence Summarization with Guidance of Selective Multimodal
  Reference
  Multimodal abstractive summarization with sentence output is to generate a
textual summary given a multimodal triad -- sentence, image and audio, which
has been proven to improve users satisfaction and convenient our life. Existing
approaches mainly focus on the enhancement of multimodal fusion, while ignoring
the unalignment among multiple inputs and the emphasis of different segments in
feature, which has resulted in the superfluity of multimodal interaction. To
alleviate these problems, we propose a Multimodal Hierarchical Selective
Transformer (mhsf) model that considers reciprocal relationships among
modalities (by low-level cross-modal interaction module) and respective
characteristics within single fusion feature (by high-level selective routing
module). In details, it firstly aligns the inputs from different sources and
then adopts a divide and conquer strategy to highlight or de-emphasize
multimodal fusion representation, which can be seen as a sparsely feed-forward
model - different groups of parameters will be activated facing different
segments in feature. We evaluate the generalism of proposed mhsf model with the
pre-trained+fine-tuning and fresh training strategies. And Further experimental
results on MSMO demonstrate that our model outperforms SOTA baselines in terms
of ROUGE, relevance scores and human evaluation.
