  Predictive modeling uncovers knowledge and insights regarding a hypothesized
data generating mechanism (DGM). Results from different studies on a complex
DGM, derived from different data sets, and using complicated models and
algorithms, are hard to quantitatively compare due to random noise and
statistical uncertainty in model results. This has been one of the main
contributors to the replication crisis in the behavioral sciences.
  The contribution of this paper is to apply prediction scoring to the problem
of comparing two studies, such as can arise when evaluating replications or
competing evidence.
  We examine the role of predictive models in quantitatively assessing
agreement between two datasets that are assumed to come from two distinct DGMs.
We formalize a distance between the DGMs that is estimated using cross
validation. We argue that the resulting prediction scores depend on the
predictive models created by cross validation. In this sense, the prediction
scores measure the distance between DGMs, along the dimension of the particular
predictive model. Using human behavior data from experimental economics, we
demonstrate that prediction scores can be used to evaluate preregistered
hypotheses and provide insights comparing data from different populations and
settings. We examine the asymptotic behavior of the prediction scores using
simulated experimental data and demonstrate that leveraging competing
predictive models can reveal important differences between underlying DGMs. Our
proposed cross-validated prediction scores are capable of quantifying
differences between unobserved data generating mechanisms and allow for the
validation and assessment of results from complex models.
