  Abstractive text summarization is one of the areas influenced by the
emergence of pre-trained language models. Current pre-training works in
abstractive summarization give more points to the summaries with more words in
common with the main text and pay less attention to the semantic similarity
between generated sentences and the original document. We propose ARMAN, a
Transformer-based encoder-decoder model pre-trained with three novel objectives
to address this issue. In ARMAN, salient sentences from a document are selected
according to a modified semantic score to be masked and form a pseudo summary.
To summarize more accurately and similar to human writing patterns, we applied
modified sentence reordering. We evaluated our proposed models on six
downstream Persian summarization tasks. Experimental results show that our
proposed model achieves state-of-the-art performance on all six summarization
tasks measured by ROUGE and BERTScore. Our models also outperform prior works
in textual entailment, question paraphrasing, and multiple choice question
answering. Finally, we established a human evaluation and show that using the
semantic score significantly improves summarization results.
