<p>A good practice is to provide <code>yml</code> file along with your <code>Snakefile</code>. This file should contain all the enviroment definitions that are needed to run your pipeline. The minimum content of this file would be to install <code>snakemake</code> itself. Any other program used in your pipeline should be included as well. For reproducibility version numbers must be used.</p>

<p>So, if someone would like to use your pipeline, (s)he first creates a new enviroment based on your definition:</p>

<pre><code>$ conda env create -f my_pipeline.yml
</code></pre>

<p>Afterwards activate this environment and run the pipeline:</p>

<pre><code><span class="math-container">$ conda activate my_pipeline
$</span> snakemake
</code></pre>

<p>The <code>conda</code> directive in your <code>Snakefile</code> is only neccessary if you need to use program which dependencies cannot be resolved within the main environment. If you start snakemake with the <code>--use-conda</code> parameter, snakemake will create a new environment based on the provided definition file and activate it for the rule.</p>

<blockquote>
  <p>Especially since it is pulled down every time someone analyses data (sometime multiple times a day).</p>
</blockquote>

<p>The environment creation is only done once. If you run the pipeline a second time <code>snakemake</code> can activate the environment diretcly without downloading and installing something again.</p>

<p>The <a href="https://www.biostars.org/p/335903/" rel="nofollow noreferrer">tutorial</a> I wrote some time ago on biostars about <code>snakemake</code> and <code>conda</code> might be useful for you.</p>

CC BY-SA 4.0
2019-02-01T08:27:18.420Z
Ah, sounds good, I see that "run:" parts also don't support a conda env, so you really need the main env to have everything as well. I'm testing this now, thanx. Will mark as accepted answer when everything works.
Freek
CC BY-SA 4.0
2019-02-01T08:21:12.453Z
2019-02-01T08:50:45.590Z
2019-02-01T08:50:45.590Z
finswimmer
<p>Inside your Snakefile you can provide the same <code>conda:</code> directive to each rule. If you install your pipeline in a central location and have everyone use the same conda when running it (hint: use a wrapper script that sets the path to conda) then everyone will be using the same environment. This has the added benefit that you can make this readable to everyone, but disallow them to make any modifications (so users can't mess things up).</p>

<p>The tack we've taken is to have a basic conda env with snakemake and little else and then separate envs for various pipelines (and versions thereof if software versions in your YAML files change) or parts thereof, in cases where you have rules using tools with conflicting dependencies. This ends up being fairly convenient for more complicated pipelines, though if you can get away with just a single environment then finswimmer's answer is simpler.</p>

<p>Note that all of this should work with any scheduler, since snakemake will insert the environment activation stuff in the script it submits to your scheduler.</p>

<p>As an aside, this is exactly how <a href="https://snakepipes.readthedocs.io/en/latest/" rel="nofollow noreferrer">snakePipes</a> works.</p>

CC BY-SA 4.0
2019-02-01T08:42:23.687Z
2019-02-01T08:42:23.687Z
Devon Ryan
<p>Using only the <code>--use-conda</code> directive, snakemake copies environments somewhere in the local <code>.snakemake</code> directory. This is repeated for each directory you run it in and replicates all environments effectively for each dataset.</p>

<p>You can however tell snakemake to use a shared environment location using <code>--conda-prefix</code> as described <a href="https://snakemake.readthedocs.io/en/stable/executing/cli.html#CONDA" rel="nofollow noreferrer">here</a>:</p>

<blockquote>
  <p>Specify a directory in which the ‘conda’ and ‘conda-archive’
  directories are created. These are used to store conda environments
  and their archives, respectively. If not supplied, the value is set to
  the ‘.snakemake’ directory relative to the invocation directory. If
  supplied, the –use-conda flag must also be set. The value may be given
  as a relative path, which will be extrapolated to the invocation
  directory, or as an absolute path.</p>
</blockquote>

<p>This way you could only download/copy the environment once and share it, e.g. this way:</p>

<pre><code>snakemake --use-conda --conda-prefix /path/to/shared/environment
</code></pre>

CC BY-SA 4.0
2019-02-05T10:25:46.310Z
Yes, this is what I was looking for, great, thanx. In this way I can still create multiple environment but they are created once and available to everybody.
Freek
CC BY-SA 4.0
2020-04-07T18:48:16.627Z
`--conda` should be replaced by `--use-conda`. Former is not supported by snakemake [as far as I could tell](https://github.com/snakemake/snakemake/blob/master/snakemake/__init__.py).
Manavalan Gajapathy
CC BY-SA 4.0
2020-04-08T06:46:27.327Z
Your are absolutely right, this has probably changed in new versions. I've replaced it and also the link was updated. Thanks for the pointer
Sebastian Müller
CC BY-SA 4.0
2019-02-01T13:07:09.493Z
2020-04-08T06:45:03.903Z
2020-04-08T06:45:03.903Z
Sebastian Müller
<p>I have been experimenting a lot lately with Snakemake, I love it. Recently I also switched to using conda (--use-conda) in the way that is advertised. However, I have some issues with it, mostly related to the way we work. I work in a lab that processes data from our sequencer but also external datasets, moreover, multiple people use my pipeline.</p>

<p>So, with respect to conda this means: currently I have 5 envs, averaging 1.5 GB per env, this takes about 20 min to set up and it's quite some data to pull down. Especially since it is pulled down every time someone analyses data (sometime multiple times a day). This feels like quite a waste of resources. Also, I fix some package versions in the yaml files, but certainly not all (this may lead to issues when repos get reshuffled I read), so I also can't guarantee the exact environment long term (say over years, dependencies may change/update).</p>

<p>So I'm considering switching to a central conda env which I version and of which I export lists of all packages installed for reproducibility reasons. This will then contain all tools.</p>

<p>My question is, is there a nice way to do this? I had issues with putting a "source activate" line in every shell command (and how to do this for Python code under "run" anyway) before, see this: <a href="https://stackoverflow.com/questions/54024862/snakemake-conda-env-parameter-is-not-taken-from-config-yaml-file">https://stackoverflow.com/questions/54024862/snakemake-conda-env-parameter-is-not-taken-from-config-yaml-file</a>.</p>

<p>So, is there a way, also when using a qsub cluster, to always use a specific conda env, for everything you do?</p>

<python><snakemake><bioconda>
Running Snakemake in one single conda env
# Running Snakemake in one single conda env

(Asked by: username_0 on 2019-01-31)

I have been experimenting a lot lately with Snakemake, I love it. Recently I also switched to using conda (--use-conda) in the way that is advertised. However, I have some issues with it, mostly related to the way we work. I work in a lab that processes data from our sequencer but also external datasets, moreover, multiple people use my pipeline.

So, with respect to conda this means: currently I have 5 envs, averaging 1.5 GB per env, this takes about 20 min to set up and it's quite some data to pull down. Especially since it is pulled down every time someone analyses data (sometime multiple times a day). This feels like quite a waste of resources. Also, I fix some package versions in the yaml files, but certainly not all (this may lead to issues when repos get reshuffled I read), so I also can't guarantee the exact environment long term (say over years, dependencies may change/update).

So I'm considering switching to a central conda env which I version and of which I export lists of all packages installed for reproducibility reasons. This will then contain all tools.

My question is, is there a nice way to do this? I had issues with putting a "source activate" line in every shell command (and how to do this for Python code under "run" anyway) before, see this: <https://stackoverflow.com/questions/54024862/snakemake-conda-env-parameter-is-not-taken-from-config-yaml-file>.

So, is there a way, also when using a qsub cluster, to always use a specific conda env, for everything you do?

**Tags:** python, snakemake, bioconda

---

## 3 answers

### Answer by username_3 on 2019-02-01. Score: 5 | Accepted answer

Using only the --use-conda directive, snakemake copies environments somewhere in the local .snakemake directory. This is repeated for each directory you run it in and replicates all environments effectively for each dataset.
You can however tell snakemake to use a shared environment location using --conda-prefix as described here:

Specify a directory in which the ‘conda’ and ‘conda-archive’
  directories are created. These are used to store conda environments
  and their archives, respectively. If not supplied, the value is set to
  the ‘.snakemake’ directory relative to the invocation directory. If
  supplied, the –use-conda flag must also be set. The value may be given
  as a relative path, which will be extrapolated to the invocation
  directory, or as an absolute path.

This way you could only download/copy the environment once and share it, e.g. this way:
snakemake --use-conda --conda-prefix /path/to/shared/environment

> **Comments:**
> **username_0**: Yes, this is what I was looking for, great, thanx. In this way I can still create multiple environment but they are created once and available to everybody.  
> **username_4**: `--conda` should be replaced by `--use-conda`. Former is not supported by snakemake [as far as I could tell](https://github.com/snakemake/snakemake/blob/master/snakemake/__init__.py).  
> **username_3**: Your are absolutely right, this has probably changed in new versions. I've replaced it and also the link was updated. Thanks for the pointer  

---

### Answer by username_1 on 2019-02-01. Score: 5

A good practice is to provide yml file along with your Snakefile. This file should contain all the enviroment definitions that are needed to run your pipeline. The minimum content of this file would be to install snakemake itself. Any other program used in your pipeline should be included as well. For reproducibility version numbers must be used.
So, if someone would like to use your pipeline, (s)he first creates a new enviroment based on your definition:
$ conda env create -f my_pipeline.yml

Afterwards activate this environment and run the pipeline:
$ conda activate my_pipeline
$ snakemake

The conda directive in your Snakefile is only neccessary if you need to use program which dependencies cannot be resolved within the main environment. If you start snakemake with the --use-conda parameter, snakemake will create a new environment based on the provided definition file and activate it for the rule.

Especially since it is pulled down every time someone analyses data (sometime multiple times a day).

The environment creation is only done once. If you run the pipeline a second time snakemake can activate the environment diretcly without downloading and installing something again.
The tutorial I wrote some time ago on biostars about snakemake and conda might be useful for you.

> **Comments:**
> **username_0**: Ah, sounds good, I see that "run:" parts also don't support a conda env, so you really need the main env to have everything as well. I'm testing this now, thanx. Will mark as accepted answer when everything works.  

---

### Answer by username_2 on 2019-02-01. Score: 1

Inside your Snakefile you can provide the same `conda:` directive to each rule. If you install your pipeline in a central location and have everyone use the same conda when running it (hint: use a wrapper script that sets the path to conda) then everyone will be using the same environment. This has the added benefit that you can make this readable to everyone, but disallow them to make any modifications (so users can't mess things up).

The tack we've taken is to have a basic conda env with snakemake and little else and then separate envs for various pipelines (and versions thereof if software versions in your YAML files change) or parts thereof, in cases where you have rules using tools with conflicting dependencies. This ends up being fairly convenient for more complicated pipelines, though if you can get away with just a single environment then finswimmer's answer is simpler.

Note that all of this should work with any scheduler, since snakemake will insert the environment activation stuff in the script it submits to your scheduler.

As an aside, this is exactly how [snakePipes](https://snakepipes.readthedocs.io/en/latest/) works.


---

