  Abstractive neural summarization models have seen great improvements in
recent years, as shown by ROUGE scores of the generated summaries. But despite
these improved metrics, there is limited understanding of the strategies
different models employ, and how those strategies relate their understanding of
language. To understand this better, we run several experiments to characterize
how one popular abstractive model, the pointer-generator model of See et al.
(2017), uses its explicit copy/generation switch to control its level of
abstraction (generation) vs extraction (copying). On an extractive-biased
dataset, the model utilizes syntactic boundaries to truncate sentences that are
otherwise often copied verbatim. When we modify the copy/generation switch and
force the model to generate, only simple paraphrasing abilities are revealed
alongside factual inaccuracies and hallucinations. On an abstractive-biased
dataset, the model copies infrequently but shows similarly limited abstractive
abilities. In line with previous research, these results suggest that
abstractive summarization models lack the semantic understanding necessary to
generate paraphrases that are both abstractive and faithful to the source
document.
