  With the advent of open source software, a veritable treasure trove of
previously proprietary software development data was made available. This
opened the field of empirical software engineering research to anyone in
academia. Data that is mined from software projects, however, requires
extensive processing and needs to be handled with utmost care to ensure valid
conclusions. Since the software development practices and tools have changed
over two decades, we aim to understand the state-of-the-art research workflows
and to highlight potential challenges. We employ a systematic literature review
by sampling over one thousand papers from leading conferences and by analyzing
the 286 most relevant papers from the perspective of data workflows,
methodologies, reproducibility, and tools. We found that an important part of
the research workflow involving dataset selection was particularly problematic,
which raises questions about the generality of the results in existing
literature. Furthermore, we found a considerable number of papers provide
little or no reproducibility instructions -- a substantial deficiency for a
data-intensive field. In fact, 33% of papers provide no information on how
their data was retrieved. Based on these findings, we propose ways to address
these shortcomings via existing tools and also provide recommendations to
improve research workflows and the reproducibility of research.
