  Jupyter notebooks facilitate the bundling of executable code with its
documentation and output in one interactive environment, and they represent a
popular mechanism to document and share computational workflows. The
reproducibility of computational aspects of research is a key component of
scientific reproducibility but has not yet been assessed at scale for Jupyter
notebooks associated with biomedical publications. We address computational
reproducibility at two levels: First, using fully automated workflows, we
analyzed the computational reproducibility of Jupyter notebooks related to
publications indexed in PubMed Central. We identified such notebooks by mining
the articles full text, locating them on GitHub and re-running them in an
environment as close to the original as possible. We documented reproduction
success and exceptions and explored relationships between notebook
reproducibility and variables related to the notebooks or publications. Second,
this study represents a reproducibility attempt in and of itself, using
essentially the same methodology twice on PubMed Central over two years. Out of
27271 notebooks from 2660 GitHub repositories associated with 3467 articles,
22578 notebooks were written in Python, including 15817 that had their
dependencies declared in standard requirement files and that we attempted to
re-run automatically. For 10388 of these, all declared dependencies could be
installed successfully, and we re-ran them to assess reproducibility. Of these,
1203 notebooks ran through without any errors, including 879 that produced
results identical to those reported in the original notebook and 324 for which
our results differed from the originally reported ones. Running the other
notebooks resulted in exceptions. We zoom in on common problems, highlight
trends and discuss potential improvements to Jupyter-related workflows
associated with biomedical publications.
