<p>Another option I'm trying at the moment is the combination of <a href="https://snakemake.readthedocs.io" rel="nofollow noreferrer">snakemake</a>, <a href="https://conda.io" rel="nofollow noreferrer">conda.io</a> and <a href="https://yihui.name/knitr/" rel="nofollow noreferrer">knitr</a>.</p>

<ul>
<li><p><strong>Conda</strong> is a "Package, dependency and environment management for any language" and could be used to set up the <strong>R-environment</strong> as well as other software. Note, most bioinformatics software can be found in the related <strong>bioconda</strong> project. </p></li>
<li><p><strong>Snakemake</strong> is a  workflow management system, which can be used to automate the preprocessing of your data in a reproducible manner (also it works nicely with conda since a conda environments can be specified in the Snake-Makefile) </p></li>
<li><p>Knitr for dynamic report generation with R. Well suited to document and reproduce your R workflow into reports, ticking your <strong>code</strong> and seed <strong>boxes</strong>.</p></li>
</ul>

CC BY-SA 3.0
2017-09-10T12:15:20.227Z
how is snakemake working nicely with conda, as respect to more "traditional" systems like Makefiles?
mgalardini
CC BY-SA 3.0
2017-09-10T21:21:04.777Z
@mgalardini I read it as "is installable and works if installed via conda".
Michael Schubert
CC BY-SA 3.0
2017-09-11T15:47:17.013Z
There are dedicated directives for conda in the Snakefile. I changed the text to make it more clear. Also have a look [here](https://snakemake.readthedocs.io/en/stable/tutorial/additional_features.html#automatic-deployment-of-software-dependencies) for further info
Sebastian Müller
CC BY-SA 3.0
2017-09-09T14:52:58.333Z
2017-09-11T15:44:39.477Z
2017-09-11T15:44:39.477Z
Sebastian Müller
<p><em>checkpoint</em> is a system for ensuring you get packages of the same time / version. However, I confess that I haven't used it and thus can't vouch for it. </p>

<p>My feeling is that the rest will have to be handled outside this: code and data can be versioned in git (and dat). A whole solution might have use something like docker as others have suggested.</p>

CC BY-SA 3.0
2017-09-12T09:12:10.283Z
2017-09-12T09:12:10.283Z
agapow
<p>One way to achieve this is via <code>packrat</code>, a dependency management system developed by RStudio.</p>

<p>Setup:</p>

<p>1) <a href="https://support.rstudio.com/hc/en-us/articles/200526207-Using-Projects" rel="nofollow noreferrer">Create project</a></p>

<p>2) <a href="https://stackoverflow.com/a/26598597">Create script</a> in project directory to start RStudio with specific R version:</p>

<pre><code>export RSTUDIO_WHICH_R=/path/to/R
rstudio
</code></pre>

<p>3) <a href="https://rstudio.github.io/packrat/rstudio.html" rel="nofollow noreferrer">Bring project</a> under <code>packrat</code> control. Install bioconductor, install packages...</p>

<hr>

<p>Every time you work on the project, run the above script in project directory.
<a href="https://github.com/rstudio/packrat/issues/342#issuecomment-315059897" rel="nofollow noreferrer">https://github.com/rstudio/packrat/issues/342#issuecomment-315059897</a></p>

CC BY-SA 3.0
2017-08-31T14:06:02.127Z
2017-08-31T14:06:02.127Z
Peter
<p>You're right about trying to get the R environments to be as similar as possible, as easily as possible. It's not always necessary to have the exact environment to get the same results, but it definitely helps if you find yourself struggling to replicate results.</p>

<h1>General tips for ensuring reproducibility, not specific to R</h1>

<p>See <a href="http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003285" rel="noreferrer">Sandve <em>et al.</em>, PLoS Comp Bio, 2013</a>. They mention a number of things that you do, like recording seed values and session info. I'll highlight a few points:</p>

<ol>
<li><strong>Environment</strong>: Include a small script that lists all the necessary dependencies

<ul>
<li>This should include all version numbers of languages and packages used</li>
<li><em>Better implementation</em>: make it a script that automatically installs dependencies</li>
</ul></li>
<li><strong>Data</strong>: Share raw and/or processed data where possible

<ul>
<li>At the very least, details on how to obtain the data you used</li>
<li>This can be an accession number in a database, or a link to the source material</li>
<li><em>Better implementation</em>: again, make it a script that automatically downloads the raw data you used</li>
</ul></li>
<li><strong>Code</strong>: Comment and document your code

<ul>
<li>People (yourself included) need to understand what you've done and why</li>
<li>Additionally, your work may be reproducible, but it may not be correct. If you (or anyone else) attempts to fix a problem in your code, they will have a difficult time doing so if they can't understand what your code is doing</li>
</ul></li>
<li><strong>Code</strong>: Use version control

<ul>
<li>Services like GitHub and BitBucket make it simple to share your exact work with collaborators</li>
<li>Version control also ensures that you have a history of all the changes done to your work</li>
</ul></li>
<li>No manual manipulation of data

<ul>
<li>If something is done to your data, write it down</li>
</ul></li>
</ol>

<h1>R-specific tips</h1>

<ol>
<li>The <code>packrat</code> package, <a href="https://bioinformatics.stackexchange.com/a/2432/240">as you mentioned in your answer</a></li>
<li>Using a number of custom functions for analyzing your data? Compile those functions into an R package

<ul>
<li>Allows others to easily load/unload the exact functions you used through tools like CRAN, BioConductor, or <code>devtools::install_github()</code></li>
<li>Makes it easy to see whether these functions are working as intended, since they are separated from other parts of the analysis</li>
</ul></li>
<li>Make unit tests for your work with the <code>testthat</code> package

<ul>
<li>Helps ensure that your code is working as intended, and gives others confidence in your work as well</li>
<li>Write the tests using <code>testthat::test_that()</code>, and check that they pass using <code>R CMD check</code> or <code>devtools::check()</code></li>
<li>This works especially well with R packages</li>
<li>This also works especially well with <a href="https://travis-ci.com" rel="noreferrer">TravisCI</a>, since you can automatically have your code get tested after you change it and make a commit</li>
</ul></li>
<li>Consider <a href="https://docker.com" rel="noreferrer">Docker</a> images for your environment

<ul>
<li>This bundles everything about the R environment together into one easily executable image (and it's not just limited to R)</li>
<li>These images can be easily shared through <a href="https://hub.docker.com/" rel="noreferrer">Docker Hub</a></li>
<li><strong>Note</strong> this might not always be the best solution, since few people know Docker and it's not always simple to use. Docker also requires admin privileges, which may not be possible for work on computer clusters</li>
<li>There is lots of current discussion about Docker in science. See <a href="https://arxiv.org/pdf/1410.0846.pdf" rel="noreferrer">Boettiger, 2014</a>, for an example, using R</li>
</ul></li>
</ol>

CC BY-SA 3.0
2017-09-01T07:18:18.657Z
If you use Docker, Bioconductor has [images too](https://www.bioconductor.org/help/docker/).
llrs
CC BY-SA 3.0
2017-08-31T15:35:45.683Z
2017-08-31T15:35:45.683Z
James Hawley
<p>I'm looking for a convenient and reliable way to make an R analysis reproducible, either at different times or across collaborators.</p>

<p>Listing the package versions or a <code>sessionInfo()</code> output is not very helpful, as it leaves the work of re-creating the environment entirely to you.</p>

<p>As I see, in order to make analysis reproducible, the following has to be the same:</p>

<ul>
<li>the <strong>R environment</strong> (R, repos (Bioconductor), packages, and maybe RStudio)</li>
<li><strong>data</strong> </li>
<li><strong>code</strong></li>
<li><strong>seed</strong> for random processes: <code>set.seed()</code></li>
</ul>

<p>I'll post an answer, but would like to hear about other options.</p>

CC BY-SA 3.0
2017-09-09T16:08:29.407Z
tl;dr: **There is currently *no way* to accomplish this in R.** packrat comes close but ultimately falls short, because it only supports loading *one version* per package, but different packages might share the same dependency, but require different versions. R needs something like Ruby’s Bundler. I’m planning to work on it, but it won’t work with packages due to fundamental architectural limitations.
Konrad Rudolph
<r><reproducibility><workflow-management>
Ultimate reproducibility in R?
# Ultimate reproducibility in R?

(Asked by: username_0 on 2017-08-31)

I'm looking for a convenient and reliable way to make an R analysis reproducible, either at different times or across collaborators.

Listing the package versions or a `sessionInfo()` output is not very helpful, as it leaves the work of re-creating the environment entirely to you.

As I see, in order to make analysis reproducible, the following has to be the same:

* the **R environment** (R, repos (Bioconductor), packages, and maybe RStudio)
* **data**
* **code**
* **seed** for random processes: `set.seed()`

I'll post an answer, but would like to hear about other options.

**Tags:** r, reproducibility, workflow-management
> **Comments:**
> **username_1**: tl;dr: **There is currently *no way* to accomplish this in R.** packrat comes close but ultimately falls short, because it only supports loading *one version* per package, but different packages might share the same dependency, but require different versions. R needs something like Ruby’s Bundler. I’m planning to work on it, but it won’t work with packages due to fundamental architectural limitations.  

---

## 4 answers

### Answer by username_6 on 2017-08-31. Score: 16 | Accepted answer

You're right about trying to get the R environments to be as similar as possible, as easily as possible. It's not always necessary to have the exact environment to get the same results, but it definitely helps if you find yourself struggling to replicate results.

# General tips for ensuring reproducibility, not specific to R

See [Sandve *et al.*, PLoS Comp Bio, 2013](http://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1003285). They mention a number of things that you do, like recording seed values and session info. I'll highlight a few points:

1. **Environment**: Include a small script that lists all the necessary dependencies
   * This should include all version numbers of languages and packages used
   * *Better implementation*: make it a script that automatically installs dependencies
2. **Data**: Share raw and/or processed data where possible
   * At the very least, details on how to obtain the data you used
   * This can be an accession number in a database, or a link to the source material
   * *Better implementation*: again, make it a script that automatically downloads the raw data you used
3. **Code**: Comment and document your code
   * People (yourself included) need to understand what you've done and why
   * Additionally, your work may be reproducible, but it may not be correct. If you (or anyone else) attempts to fix a problem in your code, they will have a difficult time doing so if they can't understand what your code is doing
4. **Code**: Use version control
   * Services like GitHub and BitBucket make it simple to share your exact work with collaborators
   * Version control also ensures that you have a history of all the changes done to your work
5. No manual manipulation of data
   * If something is done to your data, write it down

# R-specific tips

1. The `packrat` package, [as you mentioned in your answer](https://bioinformatics.stackexchange.com/a/2432/240)
2. Using a number of custom functions for analyzing your data? Compile those functions into an R package
   * Allows others to easily load/unload the exact functions you used through tools like CRAN, BioConductor, or `devtools::install_github()`
   * Makes it easy to see whether these functions are working as intended, since they are separated from other parts of the analysis
3. Make unit tests for your work with the `testthat` package
   * Helps ensure that your code is working as intended, and gives others confidence in your work as well
   * Write the tests using `testthat::test_that()`, and check that they pass using `R CMD check` or `devtools::check()`
   * This works especially well with R packages
   * This also works especially well with [TravisCI](https://travis-ci.com), since you can automatically have your code get tested after you change it and make a commit
4. Consider [Docker](https://docker.com) images for your environment
   * This bundles everything about the R environment together into one easily executable image (and it's not just limited to R)
   * These images can be easily shared through [Docker Hub](https://hub.docker.com/)
   * **Note** this might not always be the best solution, since few people know Docker and it's not always simple to use. Docker also requires admin privileges, which may not be possible for work on computer clusters
   * There is lots of current discussion about Docker in science. See [Boettiger, 2014](https://arxiv.org/pdf/1410.0846.pdf), for an example, using R

> **Comments:**
> **username_7**: If you use Docker, Bioconductor has [images too](https://www.bioconductor.org/help/docker/).  

---

### Answer by username_2 on 2017-09-09. Score: 5

Another option I'm trying at the moment is the combination of [snakemake](https://snakemake.readthedocs.io), [conda.io](https://conda.io) and [knitr](https://yihui.name/knitr/).

* **Conda** is a "Package, dependency and environment management for any language" and could be used to set up the **R-environment** as well as other software. Note, most bioinformatics software can be found in the related **bioconda** project.
* **Snakemake** is a workflow management system, which can be used to automate the preprocessing of your data in a reproducible manner (also it works nicely with conda since a conda environments can be specified in the Snake-Makefile)
* Knitr for dynamic report generation with R. Well suited to document and reproduce your R workflow into reports, ticking your **code** and seed **boxes**.

> **Comments:**
> **username_3**: how is snakemake working nicely with conda, as respect to more "traditional" systems like Makefiles?  
> **username_4**: @mgalardini I read it as "is installable and works if installed via conda".  
> **username_2**: There are dedicated directives for conda in the Snakefile. I changed the text to make it more clear. Also have a look [here](https://snakemake.readthedocs.io/en/stable/tutorial/additional_features.html#automatic-deployment-of-software-dependencies) for further info  

---

### Answer by username_5 on 2017-09-12. Score: 1

*checkpoint* is a system for ensuring you get packages of the same time / version. However, I confess that I haven't used it and thus can't vouch for it.

My feeling is that the rest will have to be handled outside this: code and data can be versioned in git (and dat). A whole solution might have use something like docker as others have suggested.


---

### Answer by username_0 on 2017-08-31. Score: 1

One way to achieve this is via packrat, a dependency management system developed by RStudio.
Setup:
1) Create project
2) Create script in project directory to start RStudio with specific R version:
export RSTUDIO_WHICH_R=/path/to/R
rstudio

3) Bring project under packrat control. Install bioconductor, install packages...

Every time you work on the project, run the above script in project directory.
https://github.com/rstudio/packrat/issues/342#issuecomment-315059897


---

