  Reproducibility of results is a cornerstone of the scientific method.
Scientific computing encounters two challenges when aiming for this goal.
Firstly, reproducibility should not depend on details of the runtime
environment, such as the compiler version or computing environment, so results
are verifiable by third-parties. Secondly, different versions of software code
executed in the same runtime environment should produce consistent numerical
results for physical quantities. In this manuscript, we test the feasibility of
reproducing scientific results obtained using the IllinoisGRMHD code that is
part of an open-source community software for simulation in relativistic
astrophysics, the Einstein Toolkit. We verify that numerical results of
simulating a single isolated neutron star with IllinoisGRMHD can be reproduced,
and compare them to results reported by the code authors in 2015. We use two
different supercomputers: Expanse at SDSC, and Stampede2 at TACC.
  By compiling the source code archived along with the paper on both Expanse
and Stampede2, we find that IllinoisGRMHD reproduces results published in its
announcement paper up to errors comparable to round-off level changes in
initial data parameters. We also verify that a current version of
IlliinoisGRMHD reproduces these results once we account for bug fixes which has
occurred since the original publication
