  Over the past few years, deep learning methods have been applied for a wide
range of Software Engineering (SE) tasks, including in particular for the
important task of automatically predicting and localizing faults in software.
With the rapid adoption of increasingly complex machine learning models, it
however becomes more and more difficult for scholars to reproduce the results
that are reported in the literature. This is in particular the case when the
applied deep learning models and the evaluation methodology are not properly
documented and when code and data are not shared. Given some recent -- and very
worrying -- findings regarding reproducibility and progress in other areas of
applied machine learning, the goal of this work is to analyze to what extent
the field of software engineering, in particular in the area of software fault
prediction, is plagued by similar problems. We have therefore conducted a
systematic review of the current literature and examined the level of
reproducibility of 56 research articles that were published between 2019 and
2022 in top-tier software engineering conferences. Our analysis revealed that
scholars are apparently largely aware of the reproducibility problem, and about
two thirds of the papers provide code for their proposed deep learning models.
However, it turned out that in the vast majority of cases, crucial elements for
reproducibility are missing, such as the code of the compared baselines, code
for data pre-processing or code for hyperparameter tuning. In these cases, it
therefore remains challenging to exactly reproduce the results in the current
research literature. Overall, our meta-analysis therefore calls for improved
research practices to ensure the reproducibility of machine-learning based
research.
