  Multimodal summarization (MS) aims to generate a summary from multimodal
input. Previous works mainly focus on textual semantic coverage metrics such as
ROUGE, which considers the visual content as supplemental data. Therefore, the
summary is ineffective to cover the semantics of different modalities. This
paper proposes a multi-task cross-modality learning framework (CISum) to
improve multimodal semantic coverage by learning the cross-modality interaction
in the multimodal article. To obtain the visual semantics, we translate images
into visual descriptions based on the correlation with text content. Then, the
visual description and text content are fused to generate the textual summary
to capture the semantics of the multimodal content, and the most relevant image
is selected as the visual summary. Furthermore, we design an automatic
multimodal semantics coverage metric to evaluate the performance. Experimental
results show that CISum outperforms baselines in multimodal semantics coverage
metrics while maintaining the excellent performance of ROUGE and BLEU.
