  Abstractive document summarization is usually modeled as a
sequence-to-sequence (Seq2Seq) learning problem. Unfortunately, training large
Seq2Seq based summarization models on limited supervised summarization data is
challenging. This paper presents three pre-training objectives which allow us
to pre-train a Seq2Seq based abstractive summarization model on unlabeled text.
The main idea is that, given an input text artificially constructed from a
document, a model is pre-trained to reinstate the original document. These
objectives include sentence reordering, next sentence generation, and masked
document generation, which have close relations with the abstractive document
summarization task. Experiments on two benchmark summarization datasets (i.e.,
CNN/DailyMail and New York Times) show that all three objectives can improve
performance upon baselines. Compared to models pre-trained on large-scale data
(more than 160GB), our method, with only 19GB text for pre-training, achieves
comparable results, which demonstrates its effectiveness.
