<p>@vantom, while I am not very familiar with snakemake on a whole, I do have extensive knowledge writing workflows in <a href="https://openwdl.org" rel="nofollow noreferrer">WDL</a> as well as CWL. The approach both of these languages take to overcome a similar issue that you are describing, is to use containerization through ie <a href="https://docs.docker.com/" rel="nofollow noreferrer">docker</a>. This approach allows you to bundle the specific tools you need for a specific task, making different images for different tasks. There are several large repositories of images online ie: <a href="https://hub.docker.com" rel="nofollow noreferrer">Docker Hub</a>, or more biologically focused: <a href="https://biocontainers.pro/" rel="nofollow noreferrer">Biocontainers</a></p>

<p>Looking through the documentation, it does appear that snakemake supports  <a href="https://snakemake.readthedocs.io/en/stable/snakefiles/deployment.html#running-jobs-in-containers" rel="nofollow noreferrer">running your rule from within the context of a docker image</a>. Without knowing how your compute environment is setup, I am not sure if there is any additional configuration you would need, however the documentation seems to suggest that it should be as easy as describing the name of the container. As long as you can run docker, (or an alternative like singularity) you should be able to avoid the current headache you are facing.</p>

<pre class="lang-yaml prettyprint-override"><code>rule NAME:
    input:
        "table.txt"
    output:
        "plots/myplot.pdf"
    containers:
        "docker://joseespinosa/docker-r-ggplot2"
    script:
        "scripts/plot-stuff.R"
</code></pre>

CC BY-SA 4.0
2020-04-30T20:33:29.357Z
Thank you for the tip! I'm not currently using Docker, but it's useful to know this for the future.
vantom
CC BY-SA 4.0
2020-09-29T02:25:51.670Z
For reproducibility, you should always used a tagged version of a container, that is:

`docker://joseespinosa/docker-r-ggplot2:v0.1`

If you don't it will pull/build the `:latest` tag and you run the risk your pipeline could break on an update.

See the `tags` menu in **DockerHub**; _e.g._ https://hub.docker.com/r/joseespinosa/docker-r-ggplot2/tags
Matthew J. Oldach
CC BY-SA 4.0
2020-04-27T20:29:16.153Z
2020-04-27T20:29:16.153Z
Patrick Magee
<p>Each YAML file will always create its own environment. The packages in each aren't downloaded and installed each time, if they're already installed elsewhere they're just hard-linked in. You'll want to use both <code>--use-conda</code> and <code>--conda-prefix</code>. The latter will point to the base directory in which all conda environments will be stored. This has the benefit that running the same workflow in different places won't result in a new env being created (by default snakemake will do so inside a .snakemake directory otherwise).</p>

CC BY-SA 4.0
2020-04-30T20:31:55.210Z
Wow, thank you. I had no idea that these two flags could be used together (I thought --conda-prefix could only be used with a named pre-built single environment) -- that solves the problem. I see what you mean about the hard-linking but it still ends up looking quite messy if you have all these .snakemake directories everywhere and it also seems to slow down some conda commands (like conda env list), at least for me.
vantom
CC BY-SA 4.0
2020-04-28T06:11:28.257Z
2020-04-28T06:11:28.257Z
Devon Ryan
<p>I have some snakemake pipelines that have rules which run in different environments. Is it possible to run the pipeline without having to rebuild the environments from scratch every time when run in a different directory?</p>

<p>This is the way I would normally run snakemake:</p>

<p><code>snakemake --directory &lt;my_output_dir&gt; --use-conda</code></p>

<p>My snakefile might have rules that need to use different environments:</p>

<pre><code>rule A:
   ....
   conda: envA.yml
   ....

rule B:
   ....
   conda: envB.yml
   ....
</code></pre>

<p>So if I wanted to do two different analyses with the same pipeline, is there anyway that I can point them to the same environments without having to install everything separately in each output directory?</p>

<p>The snakemake documentation makes it seem like they don't currently support this. You can either use 1) --conda-prefix when running snakemake and then point to a single env or 2) use the --use-conda directive and the env.yml files by rule, in which case it will create new envs any time you run it in a new output directory. So my case doesn't seem to be covered, but does anyone have a workaround to suggest? I often find conflicts between some of the R packages I use and python packages, so I tend to keep them in their own environments to avoid headache.</p>

<p>I'm starting to get neck-deep in conda environments at this point which is taking up space from my disk quota, but also I am bothered by the fact that I may actually want to run analysis of data from different projects with the same environments. If the environments are rebuilt at each execution, I do not know that they are the same, short of specifying the exact version info for everything I use in the env.yaml files (and even then, they could be slightly different if the repo has changed).</p>

<p>I have looked at this similar question: <a href="https://bioinformatics.stackexchange.com/questions/6914/running-snakemake-in-one-single-conda-env/6921#6921">Running Snakemake in one single conda env</a>
in which it is suggested there is a way to do such a thing, but I'm afraid I don't understand how to implement it without an example.</p>

<p>Thanks</p>

<snakemake>
Running snakemake with multiple conda environments without rebuilding environments every time
# Running snakemake with multiple conda environments without rebuilding environments every time

(Asked by: username_0 on 2020-04-21)

I have some snakemake pipelines that have rules which run in different environments. Is it possible to run the pipeline without having to rebuild the environments from scratch every time when run in a different directory?
This is the way I would normally run snakemake:
snakemake --directory  --use-conda
My snakefile might have rules that need to use different environments:
rule A:
   ....
   conda: envA.yml
   ....

rule B:
   ....
   conda: envB.yml
   ....

So if I wanted to do two different analyses with the same pipeline, is there anyway that I can point them to the same environments without having to install everything separately in each output directory?
The snakemake documentation makes it seem like they don't currently support this. You can either use 1) --conda-prefix when running snakemake and then point to a single env or 2) use the --use-conda directive and the env.yml files by rule, in which case it will create new envs any time you run it in a new output directory. So my case doesn't seem to be covered, but does anyone have a workaround to suggest? I often find conflicts between some of the R packages I use and python packages, so I tend to keep them in their own environments to avoid headache.
I'm starting to get neck-deep in conda environments at this point which is taking up space from my disk quota, but also I am bothered by the fact that I may actually want to run analysis of data from different projects with the same environments. If the environments are rebuilt at each execution, I do not know that they are the same, short of specifying the exact version info for everything I use in the env.yaml files (and even then, they could be slightly different if the repo has changed).
I have looked at this similar question: Running Snakemake in one single conda env
in which it is suggested there is a way to do such a thing, but I'm afraid I don't understand how to implement it without an example.
Thanks

**Tags:** snakemake

---

## 2 answers

### Answer by username_3 on 2020-04-28. Score: 4 | Accepted answer

Each YAML file will always create its own environment. The packages in each aren't downloaded and installed each time, if they're already installed elsewhere they're just hard-linked in. You'll want to use both `--use-conda` and `--conda-prefix`. The latter will point to the base directory in which all conda environments will be stored. This has the benefit that running the same workflow in different places won't result in a new env being created (by default snakemake will do so inside a .snakemake directory otherwise).

> **Comments:**
> **username_0**: Wow, thank you. I had no idea that these two flags could be used together (I thought --conda-prefix could only be used with a named pre-built single environment) -- that solves the problem. I see what you mean about the hard-linking but it still ends up looking quite messy if you have all these .snakemake directories everywhere and it also seems to slow down some conda commands (like conda env list), at least for me.  

---

### Answer by username_1 on 2020-04-27. Score: 1

@vantom, while I am not very familiar with snakemake on a whole, I do have extensive knowledge writing workflows in WDL as well as CWL. The approach both of these languages take to overcome a similar issue that you are describing, is to use containerization through ie docker. This approach allows you to bundle the specific tools you need for a specific task, making different images for different tasks. There are several large repositories of images online ie: Docker Hub, or more biologically focused: Biocontainers
Looking through the documentation, it does appear that snakemake supports  running your rule from within the context of a docker image. Without knowing how your compute environment is setup, I am not sure if there is any additional configuration you would need, however the documentation seems to suggest that it should be as easy as describing the name of the container. As long as you can run docker, (or an alternative like singularity) you should be able to avoid the current headache you are facing.
rule NAME:
    input:
        "table.txt"
    output:
        "plots/myplot.pdf"
    containers:
        "docker://joseespinosa/docker-r-ggplot2"
    script:
        "scripts/plot-stuff.R"

> **Comments:**
> **username_0**: Thank you for the tip! I'm not currently using Docker, but it's useful to know this for the future.  
> **username_2**: For reproducibility, you should always used a tagged version of a container, that is:  `docker://joseespinosa/docker-r-ggplot2:v0.1`  If you don't it will pull/build the `:latest` tag and you run the risk your pipeline could break on an update.  See the `tags` menu in **DockerHub**; _e.g._ https://hub.docker.com/r/joseespinosa/docker-r-ggplot2/tags  

---

