  Research must be reproducible in order to make an impact on science and to
contribute to the body of knowledge in our field. Yet studies have shown that
70% of research from academic labs cannot be reproduced. In software
engineering, and more specifically requirements engineering (RE), reproducible
research is rare, with datasets not always available or methods not fully
described. This lack of reproducible research hinders progress, with
researchers having to replicate an experiment from scratch. A researcher
starting out in RE has to sift through conference papers, finding ones that are
empirical, then must look through the data available from the empirical paper
(if any) to make a preliminary determination if the paper can be reproduced.
This paper addresses two parts of that problem, identifying RE papers and
identifying empirical papers within the RE papers. Recent RE and empirical
conference papers were used to learn features and to build an automatic
classifier to identify RE and empirical papers. We introduce the Empirical
Requirements Research Classifier (ERRC) method, which uses natural language
processing and machine learning to perform supervised classification of
conference papers. We compare our method to a baseline keyword-based approach.
To evaluate our approach, we examine sets of papers from the IEEE Requirements
Engineering conference and the IEEE International Symposium on Software Testing
and Analysis. We found that the ERRC method performed better than the baseline
method in all but a few cases.
