  Background: Many published machine learning studies are irreproducible.
Issues with methodology and not properly accounting for variation introduced by
the algorithm themselves or their implementations are attributed as the main
contributors to the irreproducibility.Problem: There exist no theoretical
framework that relates experiment design choices to potential effects on the
conclusions. Without such a framework, it is much harder for practitioners and
researchers to evaluate experiment results and describe the limitations of
experiments. The lack of such a framework also makes it harder for independent
researchers to systematically attribute the causes of failed reproducibility
experiments. Objective: The objective of this paper is to develop a framework
that enable applied data science practitioners and researchers to understand
which experiment design choices can lead to false findings and how and by this
help in analyzing the conclusions of reproducibility experiments. Method: We
have compiled an extensive list of factors reported in the literature that can
lead to machine learning studies being irreproducible. These factors are
organized and categorized in a reproducibility framework motivated by the
stages of the scientific method. The factors are analyzed for how they can
affect the conclusions drawn from experiments. A model comparison study is used
as an example. Conclusion: We provide a framework that describes machine
learning methodology from experimental design decisions to the conclusions
inferred from them.
