  Scientific extreme summarization (TLDR) aims to form ultra-short summaries of
scientific papers. Previous efforts on curating scientific TLDR datasets failed
to scale up due to the heavy human annotation and domain expertise required. In
this paper, we propose a simple yet effective approach to automatically
extracting TLDR summaries for scientific papers from their citation texts.
Based on the proposed approach, we create a new benchmark CiteSum without human
annotation, which is around 30 times larger than the previous human-curated
dataset SciTLDR. We conduct a comprehensive analysis of CiteSum, examining its
data characteristics and establishing strong baselines. We further demonstrate
the usefulness of CiteSum by adapting models pre-trained on CiteSum (named
CITES) to new tasks and domains with limited supervision. For scientific
extreme summarization, CITES outperforms most fully-supervised methods on
SciTLDR without any fine-tuning and obtains state-of-the-art results with only
128 examples. For news extreme summarization, CITES achieves significant gains
on XSum over its base model (not pre-trained on CiteSum), e.g., +7.2 ROUGE-1
zero-shot performance and state-of-the-art few-shot performance. For news
headline generation, CITES performs the best among unsupervised and zero-shot
methods on Gigaword. Our dataset and code can be found at
https://github.com/morningmoni/CiteSum.
