  Source Code Summarization is the task of writing short, natural language
descriptions of source code. The main use for these descriptions is in software
documentation e.g. the one-sentence Java method descriptions in JavaDocs. Code
summarization is rapidly becoming a popular research problem, but progress is
restrained due to a lack of suitable datasets. In addition, a lack of community
standards for creating datasets leads to confusing and unreproducible research
results -- we observe swings in performance of more than 33% due only to
changes in dataset design. In this paper, we make recommendations for these
standards from experimental results. We release a dataset based on prior work
of over 2.1m pairs of Java methods and one sentence method descriptions from
over 28k Java projects. We describe the dataset and point out key differences
from natural language data, to guide and support future researchers.
