  Neural network models have shown excellent fluency and performance when
applied to abstractive summarization. Many approaches to neural abstractive
summarization involve the introduction of significant inductive bias,
exemplified through the use of components such as pointer-generator
architectures, coverage, and partially extractive procedures, designed to mimic
the process by which humans summarize documents. We show that it is possible to
attain competitive performance by instead directly viewing summarization as a
language modeling problem and effectively leveraging transfer learning. We
introduce a simple procedure built upon decoder-transformers to obtain highly
competitive ROUGE scores for summarization performance using a language
modeling loss alone, with no beam-search or other decoding-time optimization,
and instead relying on efficient nucleus sampling and greedy decoding.
