<p>The solutions are either, upgrade <code>snakemake</code> or just take all wildcards out of <code>cluster.json</code> file and write them directly to snakemake execution command :</p>

<pre><code>snakemake -p --jobs 10 --cluster-config cluster.json --cluster "bsub -J "{rule}" -q {cluster.queue} -n {threads} -M {resources.M} -R {cluster.one_host} -o {"logs/cluster/{rule}.{wildcards}.out" -e "logs/cluster/{rule}.{wildcards}.err""
</code></pre>

<p>However, I ended up instead of using of json config file in writing a simple bash wrapper <code>snakemake_cluster.sh</code> that remembers the execution command for me in more readable format :</p>

<pre><code>#!/usr/bin/env bash

snakemake $@ -p --jobs 10 --cluster "bsub \
-J {rule} \
-q bgee \
-n {threads} \
-M {resources.M} \
-R \"span[hosts=1]\" \
-o logs/cluster/{rule}.{wildcards}.out \
-e logs/cluster/{rule}.{wildcards}.err"
</code></pre>

<p>Edit : All spaces after <code>--cluster</code> command got to be escaped, otherwise it would add arguments to <code>snakemake</code> not to <code>bsub</code>.</p>

<p>Edit 2 : <code>$@</code> means all arguments, therefore it can be either targets, flags or combinations (targets got to be specified first). So, <code>snakemake_cluster.sh my_file</code> builds <code>my_file</code>, <code>snakemake_cluster.sh --quiet</code> builds default target quietly, and <code>snakemake_cluster.sh my_file --quiet</code> builds <code>my_file</code> quietly.</p>

CC BY-SA 3.0
2017-11-23T09:46:16.770Z
2017-11-23T11:57:16.147Z
2017-11-23T11:57:16.147Z
Kamil S Jaron
<p>As of the most recent snakemake (v5.10.0) the recommended way of defining resources is through a <a href="https://snakemake.readthedocs.io/en/stable/executing/cli.html#profiles" rel="nofollow noreferrer">profile</a>.  </p>

<p>It seems you are trying to submit a job on an LSF cluster system - there is a snakemake <a href="https://github.com/Snakemake-Profiles/snakemake-lsf" rel="nofollow noreferrer">profile for LSF</a> to save you having to do this work yourself.</p>

<p>It reduces your snakemake submission to something like</p>

<pre class="lang-sh prettyprint-override"><code>snakemake --profile "lsf"
</code></pre>

CC BY-SA 4.0
2020-03-15T13:38:41.023Z
Luckily I don't have to deal with `lsf` anymore, but thanks anyway, I will take a look at this feature.
Kamil S Jaron
CC BY-SA 4.0
2020-03-14T14:03:20.063Z
2020-03-14T14:03:20.063Z
Michael Hall
<p>This feature was added in version 3.6.0. Look at <a href="http://snakemake.readthedocs.io/en/stable/project_info/history.html" rel="nofollow noreferrer">release notes</a>.</p>

<p><code>{wildcards}</code> normally expands to a something that looks vaguely like a dictionary. In other words, something like <code>key1=value1,key2=value2</code>. Whether you really want to use that as a log file name is up to you.</p>

<p>Let's use a very contrived example to demonstrate this. Here is the <code>Snakefile</code>:</p>

<pre><code>rule all:
    input: "input1.human.txt"

rule blah:
    output: "{sample}.{reference}.txt"
    shell: "touch {output}"
</code></pre>

<p>I'm using your <code>cluster.json</code>, so with that let's use a contrived cluster submission program:</p>

<pre><code>snakemake --cluster-config cluster.json --cluster "echo 'rule.wildcards: {rule}.{wildcards} cluster: {cluster} -o {cluster.output} -e {cluster.error} -n {cluster.name}' &gt;&gt; log &amp;&amp; " --jobs 1
</code></pre>

<p>If you run that and look in <code>log</code>, you can see that <code>{rule}</code> is expanded as expected, <code>{wildcards}</code> is expanded to <code>sample=input1,reference=human</code>. The <code>output</code> and <code>error</code> parts from <code>cluster.json</code> are expanded correctly for me. Below are the contents of <code>log</code>:</p>

<pre><code>rule.wildcards: blah.sample=input1,reference=human cluster: one_host="span[hosts=1]",output=logs/cluster/blah.sample=input1,reference=human.out,name=blah.sample=input1,reference=human,queue=bgee,error=logs/cluster/blah.sample=input1,reference=human.err -o logs/cluster/blah.sample=input1,reference=human.out -e logs/cluster/blah.sample=input1,reference=human.err -n blah.sample=input1,reference=human
</code></pre>

<p>I'm not a big fan of using <code>{wildcards}</code> in paths unless you're doing a parameter sweep, but to each their own. Note that the above results are from snakemake-4.3.1.</p>

CC BY-SA 3.0
2017-11-22T22:51:55.457Z
2017-11-23T09:32:34.083Z
2017-11-23T09:32:34.083Z
Devon Ryan
<p>I am trying to run <code>snakemake</code> version <code>3.5.4</code> on my cluster, where jobs are partially defined by a common template build according <a href="http://snakemake.readthedocs.io/en/stable/snakefiles/configuration.html#cluster-configuration" rel="nofollow noreferrer">manual</a> and also defined though rule-specific parameters. But for some reason cluster configuration is not expanding wildcards (and command does). When I run</p>

<pre><code>snakemake data/genome.fa.gz.bwt -p --jobs 10 --cluster-config cluster.json --cluster "bsub -J "{rule}.{wildcards}" -q {cluster.queue} -n {threads} -M {resources.M} -R {cluster.one_host} -o {cluster.output} -e {cluster.error}"
</code></pre>

<p><code>-J "{rule}.{wildcards}" -q {cluster.queue} -n {threads} -M {resources.M} -R {cluster.one_host}</code> expands as expected. But <code>-o {cluster.output} -e {cluster.error}</code> is replaced by literally <code>logs/cluster/{rule}.{wildcards}.out</code> and <code>logs/cluster/{rule}.{wildcards}.err</code>, i.e. there is no expansion in config file (the config file is attached bellow).</p>

<p>I am puzzled since exactly this lines are in the documentation. What am I doing wrong? Why <code>snakemake</code> is not expanding wildcards in the config file?</p>

<p>Here is my <code>cluster.json</code></p>

<pre><code>{
    "__default__" :
    {
        "queue"    : "bgee",
        "one_host" : "\"span[hosts=1]\"",
        "name"      : "{rule}.{wildcards}",
        "output"    : "logs/cluster/{rule}.{wildcards}.out",
        "error"     : "logs/cluster/{rule}.{wildcards}.err"
    },
}
</code></pre>

<p>Here is relevant part of <code>Snakefile</code> :</p>

<pre><code>rule index_reference :
        threads: 1
        resources: M=20000000
        input : "data/{reference}/genome.fa.gz"
        output : "data/{reference}/genome.fa.gz.bwt"
        shell : "scripts/index_genome.sh {input}"
</code></pre>

<snakemake>
How to specify resources for cluster in snakemake
# How to specify resources for cluster in snakemake

(Asked by: username_0 on 2017-11-22)

I am trying to run snakemake version 3.5.4 on my cluster, where jobs are partially defined by a common template build according manual and also defined though rule-specific parameters. But for some reason cluster configuration is not expanding wildcards (and command does). When I run
snakemake data/genome.fa.gz.bwt -p --jobs 10 --cluster-config cluster.json --cluster "bsub -J "{rule}.{wildcards}" -q {cluster.queue} -n {threads} -M {resources.M} -R {cluster.one_host} -o {cluster.output} -e {cluster.error}"

-J "{rule}.{wildcards}" -q {cluster.queue} -n {threads} -M {resources.M} -R {cluster.one_host} expands as expected. But -o {cluster.output} -e {cluster.error} is replaced by literally logs/cluster/{rule}.{wildcards}.out and logs/cluster/{rule}.{wildcards}.err, i.e. there is no expansion in config file (the config file is attached bellow).
I am puzzled since exactly this lines are in the documentation. What am I doing wrong? Why snakemake is not expanding wildcards in the config file?
Here is my cluster.json
{
    "__default__" :
    {
        "queue"    : "bgee",
        "one_host" : "\"span[hosts=1]\"",
        "name"      : "{rule}.{wildcards}",
        "output"    : "logs/cluster/{rule}.{wildcards}.out",
        "error"     : "logs/cluster/{rule}.{wildcards}.err"
    },
}

Here is relevant part of Snakefile :
rule index_reference :
        threads: 1
        resources: M=20000000
        input : "data/{reference}/genome.fa.gz"
        output : "data/{reference}/genome.fa.gz.bwt"
        shell : "scripts/index_genome.sh {input}"

**Tags:** snakemake

---

## 3 answers

### Answer by username_2 on 2017-11-22. Score: 5 | Accepted answer

This feature was added in version 3.6.0. Look at release notes.
{wildcards} normally expands to a something that looks vaguely like a dictionary. In other words, something like key1=value1,key2=value2. Whether you really want to use that as a log file name is up to you.
Let's use a very contrived example to demonstrate this. Here is the Snakefile:
rule all:
    input: "input1.human.txt"

rule blah:
    output: "{sample}.{reference}.txt"
    shell: "touch {output}"

I'm using your cluster.json, so with that let's use a contrived cluster submission program:
snakemake --cluster-config cluster.json --cluster "echo 'rule.wildcards: {rule}.{wildcards} cluster: {cluster} -o {cluster.output} -e {cluster.error} -n {cluster.name}' >> log && " --jobs 1

If you run that and look in log, you can see that {rule} is expanded as expected, {wildcards} is expanded to sample=input1,reference=human. The output and error parts from cluster.json are expanded correctly for me. Below are the contents of log:
rule.wildcards: blah.sample=input1,reference=human cluster: one_host="span[hosts=1]",output=logs/cluster/blah.sample=input1,reference=human.out,name=blah.sample=input1,reference=human,queue=bgee,error=logs/cluster/blah.sample=input1,reference=human.err -o logs/cluster/blah.sample=input1,reference=human.out -e logs/cluster/blah.sample=input1,reference=human.err -n blah.sample=input1,reference=human

I'm not a big fan of using {wildcards} in paths unless you're doing a parameter sweep, but to each their own. Note that the above results are from snakemake-4.3.1.


---

### Answer by username_0 on 2017-11-23. Score: 3

The solutions are either, upgrade snakemake or just take all wildcards out of cluster.json file and write them directly to snakemake execution command :
snakemake -p --jobs 10 --cluster-config cluster.json --cluster "bsub -J "{rule}" -q {cluster.queue} -n {threads} -M {resources.M} -R {cluster.one_host} -o {"logs/cluster/{rule}.{wildcards}.out" -e "logs/cluster/{rule}.{wildcards}.err""

However, I ended up instead of using of json config file in writing a simple bash wrapper snakemake_cluster.sh that remembers the execution command for me in more readable format :
#!/usr/bin/env bash

snakemake $@ -p --jobs 10 --cluster "bsub \
-J {rule} \
-q bgee \
-n {threads} \
-M {resources.M} \
-R \"span[hosts=1]\" \
-o logs/cluster/{rule}.{wildcards}.out \
-e logs/cluster/{rule}.{wildcards}.err"

Edit : All spaces after --cluster command got to be escaped, otherwise it would add arguments to snakemake not to bsub.
Edit 2 : $@ means all arguments, therefore it can be either targets, flags or combinations (targets got to be specified first). So, snakemake_cluster.sh my_file builds my_file, snakemake_cluster.sh --quiet builds default target quietly, and snakemake_cluster.sh my_file --quiet builds my_file quietly.


---

### Answer by username_1 on 2020-03-14. Score: 1

As of the most recent snakemake (v5.10.0) the recommended way of defining resources is through a profile.  
It seems you are trying to submit a job on an LSF cluster system - there is a snakemake profile for LSF to save you having to do this work yourself.
It reduces your snakemake submission to something like
snakemake --profile "lsf"

> **Comments:**
> **username_0**: Luckily I don't have to deal with `lsf` anymore, but thanks anyway, I will take a look at this feature.  

---

