<blockquote>
<p>Save the different scripts with git (seems overkill)</p>
</blockquote>
<p>Whoa. I did an actual double take when reading this:<sup>1</sup> it’s the <em>opposite</em> of overkill.</p>
<p>Version controlling your scripts (using Git or something similar) is the absolute minimum, and should become completely automatic. For every new project I begin, one of the very first steps is to issue the <code>git init</code> command, and to set up a remote repository (on Github).</p>
<p>To keep track of <em>different</em> analyses, I use a combination of the following approaches:</p>
<ol>
<li>Write reusable functions/scripts and parametrise. The parameters are kept either inside the script itself (that then calls the relevant function repeatedly), or in a declarative workflow.<sup>2</sup></li>
<li>Document the alternative analysis approaches; once again, this could be a declarative workflow with different rules for alternative analyses, or a set of notebooks (via <a href="http://rmarkdown.rstudio.com/" rel="nofollow noreferrer">R Markdown</a>).</li>
<li>Have different Git branches for mutually exclusive approaches. At the end of the analysis one of these branches gets merged into the main branch, and published. If I want to publish several analysis approaches, I merge all these branches into the main branch, and use approaches (1) or (2) enable them simultaneously.</li>
</ol>
<p>In fact, I recommend creating a declarative workflow script for <em>every</em> analysis; I have found that this is the most practical, self-documenting way to run an analysis. It most closely resembles a wet-lab lab notebook. The advantage over a single R Markdown document is that rerunning just parts of the analysis can be completely automated, and dependencies in the workflow are apparent from the dependencies of the workflow rules. This is much harder in R Markdown.</p>
<p>Some time ago I create an <a href="https://github.com/klmr/example-r-analysis" rel="nofollow noreferrer">example analysis workflow</a> to show how this can be structured.</p>
<p>Regarding your other point:</p>
<blockquote>
<p>Make notes in the script itself</p>
</blockquote>
<p>“Notes” are a dangerous beasts: documentation is important, but experience shows that it’s sometimes very hard to keep documentation synchronised with the code. There is no mechanism that ensures that documentation and code actually agree. Differences between presumed analysis and actually executed analysis can become very problematic.</p>
<p>People therefore prefer using self-documenting code as much as possible; that is: writing code so that its meaning becomes immediately clear, without comments, even to somebody who hasn’t worked on the code before. Doing this well is hard and takes practice, but improves overall code quality. Once again, using a declarative workflow helps here because the dependencies between the rules are self-documenting the kind of analysis that was performed.</p>
<p>Jeff Atwood has written two seminal essays on this subject:</p>
<ul>
<li><a href="https://blog.codinghorror.com/coding-without-comments/" rel="nofollow noreferrer">Coding without comments</a>, and</li>
<li><a href="https://blog.codinghorror.com/code-tells-you-how-comments-tell-you-why/" rel="nofollow noreferrer">Code tells you how, comments tell you why</a>.</li>
</ul>
<p>They are two of the best pieces of advice on programming that I could give.</p>
<hr />
<p><sup>1</sup> To emphasise: take a look at the <a href="https://bioinformatics.stackexchange.com/revisions/754/1">edit history</a> of this answer.</p>
<p><sup>2</sup> I recommend <a href="https://docs.ropensci.org/targets/" rel="nofollow noreferrer">‘targets’</a> (for R), <a href="https://www.nextflow.io/" rel="nofollow noreferrer">Nextflow</a> or <a href="https://snakemake.readthedocs.io/en/stable/" rel="nofollow noreferrer">Snakemake</a>. I used to use (and recommend) GNU make, but <a href="https://twitter.com/klmr/status/1156949097116590080" rel="nofollow noreferrer">I no longer believe this to be an adequate tool</a>.</p>

CC BY-SA 4.0
2022-05-12T17:19:41.443Z
The question was clearly about storing scripts in separate branches, not just general git usage.
Brady Gilg
CC BY-SA 4.0
2022-05-13T08:26:04.377Z
@BradyGilg Yes, and so is my answer — in particular, the middle part.
Konrad Rudolph
CC BY-SA 3.0
2017-06-15T15:43:09.120Z
I appreciate the edit, but it *does* distort the meaning of my answer: The outrage implicit in the “What????!” is entirely intentional. I’ll change it to something else since people seem to disapprove.
Konrad Rudolph
CC BY-SA 3.0
2017-06-15T15:59:57.443Z
+1 for the emphasis on makefiles. Makefile and version control make an awesome combo for reproducibility!
Kamil S Jaron
CC BY-SA 3.0
2017-07-04T09:56:39.193Z
Maybe I should have stated that I meant branching. That seemed to be a lot of overhead for running an R function with a different parameter setting. I'd like to accept your answer too.. but can accept only one.
Peter
CC BY-SA 4.0
2017-06-15T12:47:47.027Z
2022-05-13T08:47:38.670Z
2022-05-13T08:47:38.670Z
Konrad Rudolph
<p>I quite agree with <a href="https://bioinformatics.stackexchange.com/a/754/292">this answer</a> by @Konrad Rudolph.</p>

<p>I would like to emphasis that using parametrization for your scripts is what will help you avoid multiplying branches in git. So yes, use git, but you don't necessarily have to go "overkill" creating lots of branches.</p>

<p>Then, I would command these scripts from a workflow managment tool that will somehow take care of generating the branches of your analyses. If you use Snakemake, the various options taken along the path from your data to your results will be represented by the wildcards system, and this will be visible in the structure of your folders and file names, due to the fact that Snakemake works by inferring what should be done to produce a file based on its name.</p>

<p>This of course is not an excuse for not using other documentation approaches: Comments in the Snakefile and in the scripts, README files explaining how the workflow was run, using what configuration file. Put your scripts, the Snakefile, its configuration files and the README files under version control and document again using commit messages.</p>

CC BY-SA 3.0
2017-06-15T15:01:31.443Z
2017-06-15T15:01:31.443Z
bli
<p>The main purpose of git is to version code, which usually means sequential improvement of the codebase. While it is possible to use branches for multiple variants of the software, permanent branches are traditionally used for gradual integration of new features (i.e. dev/testing/master branches). Supporting multiple independent branches requires some investment, i.e. distributing common changes among branches via merge or cherry-pick. This is hard to manage when you have more than two-three branches.</p>

<p>If you compare different methods of analysis, you probably want to compare the results between methods. Having them on separate branches makes it hard.</p>

<p>In my opinion you should integrate all methods of analysis into the master branch. To avoid copy &amp; paste, it is better to put common code in a library or an independent script. You can also specify a method as a run-time parameter of your pipeline, and create a meta-script which will execute all methods of interests.</p>

<p>Once you performed benchmarking, you shouldn't remove unused methods from you master branch. Having them is important for reproducible research, and your scripts could be used in the future for new datasets.</p>

CC BY-SA 3.0
2017-06-15T12:36:46.793Z
"You can also specify a method as a run-time parameter of your pipeline, and create a meta-script which will execute all methods of interests." This is the approach I use. Allows maximum flexibility for code reuse and keeps a straightforward record of everything that I tried.
heathobrien
CC BY-SA 3.0
2017-06-15T12:48:37.197Z
Don't forget to store the output of the code (with the sessionInfo) if you run the meta-script.R
llrs
CC BY-SA 3.0
2017-06-15T12:26:26.903Z
2017-06-15T12:26:26.903Z
Iakov Davydov
<p>I'm going through an RNA-seq pipeline in <code>R/Bioconductor</code> and want to try multiple parameters at subsequent steps, for example, running clustering with different settings, running RegressOut or not on unwanted effects etc. That's a lot of "versions", even if I don't do combinations of these steps.</p>

<p>How can I keep track of this, and my conclusions?
Not necessarily want to save the results.</p>

<ul>
<li>Save the different scripts with git (seems overkill)</li>
<li>Make notes in the script itself</li>
</ul>

<r><bioconductor><git><workflow-management><best-practice>
What are the ways to keep track of branches in the analysis?
# What are the ways to keep track of branches in the analysis?

(Asked by: username_0 on 2017-06-15)

I'm going through an RNA-seq pipeline in `R/Bioconductor` and want to try multiple parameters at subsequent steps, for example, running clustering with different settings, running RegressOut or not on unwanted effects etc. That's a lot of "versions", even if I don't do combinations of these steps.

How can I keep track of this, and my conclusions?
Not necessarily want to save the results.

* Save the different scripts with git (seems overkill)
* Make notes in the script itself

**Tags:** r, bioconductor, git, workflow-management, best-practice

---

## 3 answers

### Answer by username_5 on 2017-06-15. Score: 5 | Accepted answer

The main purpose of git is to version code, which usually means sequential improvement of the codebase. While it is possible to use branches for multiple variants of the software, permanent branches are traditionally used for gradual integration of new features (i.e. dev/testing/master branches). Supporting multiple independent branches requires some investment, i.e. distributing common changes among branches via merge or cherry-pick. This is hard to manage when you have more than two-three branches.

If you compare different methods of analysis, you probably want to compare the results between methods. Having them on separate branches makes it hard.

In my opinion you should integrate all methods of analysis into the master branch. To avoid copy & paste, it is better to put common code in a library or an independent script. You can also specify a method as a run-time parameter of your pipeline, and create a meta-script which will execute all methods of interests.

Once you performed benchmarking, you shouldn't remove unused methods from you master branch. Having them is important for reproducible research, and your scripts could be used in the future for new datasets.

> **Comments:**
> **username_6**: "You can also specify a method as a run-time parameter of your pipeline, and create a meta-script which will execute all methods of interests." This is the approach I use. Allows maximum flexibility for code reuse and keeps a straightforward record of everything that I tried.  
> **username_7**: Don't forget to store the output of the code (with the sessionInfo) if you run the meta-script.R  

---

### Answer by username_1 on 2017-06-15. Score: 9

> Save the different scripts with git (seems overkill)

Whoa. I did an actual double take when reading this:1 it’s the *opposite* of overkill.

Version controlling your scripts (using Git or something similar) is the absolute minimum, and should become completely automatic. For every new project I begin, one of the very first steps is to issue the `git init` command, and to set up a remote repository (on Github).

To keep track of *different* analyses, I use a combination of the following approaches:

1. Write reusable functions/scripts and parametrise. The parameters are kept either inside the script itself (that then calls the relevant function repeatedly), or in a declarative workflow.2
2. Document the alternative analysis approaches; once again, this could be a declarative workflow with different rules for alternative analyses, or a set of notebooks (via [R Markdown](http://rmarkdown.rstudio.com/)).
3. Have different Git branches for mutually exclusive approaches. At the end of the analysis one of these branches gets merged into the main branch, and published. If I want to publish several analysis approaches, I merge all these branches into the main branch, and use approaches (1) or (2) enable them simultaneously.

In fact, I recommend creating a declarative workflow script for *every* analysis; I have found that this is the most practical, self-documenting way to run an analysis. It most closely resembles a wet-lab lab notebook. The advantage over a single R Markdown document is that rerunning just parts of the analysis can be completely automated, and dependencies in the workflow are apparent from the dependencies of the workflow rules. This is much harder in R Markdown.

Some time ago I create an [example analysis workflow](https://github.com/klmr/example-r-analysis) to show how this can be structured.

Regarding your other point:

> Make notes in the script itself

“Notes” are a dangerous beasts: documentation is important, but experience shows that it’s sometimes very hard to keep documentation synchronised with the code. There is no mechanism that ensures that documentation and code actually agree. Differences between presumed analysis and actually executed analysis can become very problematic.

People therefore prefer using self-documenting code as much as possible; that is: writing code so that its meaning becomes immediately clear, without comments, even to somebody who hasn’t worked on the code before. Doing this well is hard and takes practice, but improves overall code quality. Once again, using a declarative workflow helps here because the dependencies between the rules are self-documenting the kind of analysis that was performed.

Jeff Atwood has written two seminal essays on this subject:

* [Coding without comments](https://blog.codinghorror.com/coding-without-comments/), and
* [Code tells you how, comments tell you why](https://blog.codinghorror.com/code-tells-you-how-comments-tell-you-why/).

They are two of the best pieces of advice on programming that I could give.

---

1 To emphasise: take a look at the [edit history](https://bioinformatics.stackexchange.com/revisions/754/1) of this answer.

2 I recommend [‘targets’](https://docs.ropensci.org/targets/) (for R), [Nextflow](https://www.nextflow.io/) or [Snakemake](https://snakemake.readthedocs.io/en/stable/). I used to use (and recommend) GNU make, but [I no longer believe this to be an adequate tool](https://twitter.com/klmr/status/1156949097116590080).

> **Comments:**
> **username_1**: I appreciate the edit, but it *does* distort the meaning of my answer: The outrage implicit in the “What????!” is entirely intentional. I’ll change it to something else since people seem to disapprove.  
> **username_3**: +1 for the emphasis on makefiles. Makefile and version control make an awesome combo for reproducibility!  
> **username_0**: Maybe I should have stated that I meant branching. That seemed to be a lot of overhead for running an R function with a different parameter setting. I'd like to accept your answer too.. but can accept only one.  
> **username_2**: The question was clearly about storing scripts in separate branches, not just general git usage.  
> **username_1**: @BradyGilg Yes, and so is my answer — in particular, the middle part.  

---

### Answer by username_4 on 2017-06-15. Score: 1

I quite agree with [this answer](https://bioinformatics.stackexchange.com/a/754/292) by @Konrad Rudolph.

I would like to emphasis that using parametrization for your scripts is what will help you avoid multiplying branches in git. So yes, use git, but you don't necessarily have to go "overkill" creating lots of branches.

Then, I would command these scripts from a workflow managment tool that will somehow take care of generating the branches of your analyses. If you use Snakemake, the various options taken along the path from your data to your results will be represented by the wildcards system, and this will be visible in the structure of your folders and file names, due to the fact that Snakemake works by inferring what should be done to produce a file based on its name.

This of course is not an excuse for not using other documentation approaches: Comments in the Snakefile and in the scripts, README files explaining how the workflow was run, using what configuration file. Put your scripts, the Snakefile, its configuration files and the README files under version control and document again using commit messages.


---

