  Results of simulation studies evaluating the performance of statistical
methods are often considered actionable and thus can have a major impact on the
way empirical research is implemented. However, so far there is limited
evidence about the reproducibility and replicability of statistical simulation
studies. Therefore, eight highly cited statistical simulation studies were
selected, and their replicability was assessed by teams of replicators with
formal training in quantitative methodology. The teams found relevant
information in the original publications and used it to write simulation code
with the aim of replicating the results. The primary outcome was the
feasibility of replicability based on reported information in the original
publications. Replicability varied greatly: Some original studies provided
detailed information leading to almost perfect replication of results, whereas
other studies did not provide enough information to implement any of the
reported simulations. Replicators had to make choices regarding missing or
ambiguous information in the original studies, error handling, and software
environment. Factors facilitating replication included public availability of
code, and descriptions of the data-generating procedure and methods in graphs,
formulas, structured text, and publicly accessible additional resources such as
technical reports. Replicability of statistical simulation studies was mainly
impeded by lack of information and sustainability of information sources.
Reproducibility could be achieved for simulation studies by providing open code
and data as a supplement to the publication. Additionally, simulation studies
should be transparently reported with all relevant information either in the
research paper itself or in easily accessible supplementary material to allow
for replicability.
