<p>Ok, I think the Slurm Executor is the preferred way, but there is an issue wrt my cluster (the last version does not work, but 3 version back does), I have filed an issue on GitHub: <a href="https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/157" rel="nofollow noreferrer">https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/157</a>, hope to resolve this soon, I'll put my efforts towards getting the Slurm executor working well for me.</p>
<p>FYI, it turns out that in my case I probably shouldn't send jobs to other partitions in my particular case.</p>
<p>Moreover, this code from above was a mistake, the Slurm executor does not accept this:</p>
<pre class="lang-bash prettyprint-override"><code>-o cluster_outputs/smk.{rule}.{jobid}.out -e cluster_outputs/smk.{rule}.{jobid}.err'
</code></pre>
<p>And probably for the better because once you get started with job-grouping that line does not make sense anymore (there are now multiple rules or samples per job!).</p>

CC BY-SA 4.0
2024-10-27T21:18:48.313Z
2024-10-28T07:53:00.330Z
2024-10-28T07:53:00.330Z
Freek
<p>I've been experimenting with Snakemake on a (multi-partition) Slurm cluster (Snellius, the Dutch super computer).</p>
<p>I have switched to Snakemake 8 so that I could make easier use of profiles. So I am using executors. I have tried 2 so far, snakemake-executor-plugin-cluster-generic and snakemake-executor-plugin-slurm.</p>
<p>They both have down-sides, using snakemake-executor-plugin-cluster-generic I can't seem to be able to select the partition that I want, I tried this:</p>
<pre class="lang-yaml prettyprint-override"><code>executor: cluster-generic
cluster-generic-submit-cmd:
  sbatch
    --cpus-per-task=16
    --job-name={rule}-{jobid}
    --output=cluster_outputs/{rule}/{rule}-{wildcards}-%j.out
    --parsable
restart-times: 3
max-jobs-per-second: 10
max-status-checks-per-second: 1
local-cores: 1
latency-wait: 60
jobs: 100 # Check what the max is
keep-going: True
rerun-incomplete: True
printshellcmds: True
# scheduler: greedy
use-conda: True

set-threads:
  salmon: 16

set-resources:
  fastqc:
    partition: staging
    # mem_mb: 100
</code></pre>
<p>In the submitted job I indeed see: <code>resources: tmpdir=&lt;TBD&gt;, partition=staging</code></p>
<p>I also tried:</p>
<pre class="lang-yaml prettyprint-override"><code>set-resources:
  fastqc:
    slurm_partition: staging
</code></pre>
<p>Again, I see it scrolling by under &quot;resources&quot; however, using squeue and the logs I see that the jobs are still executed on the standard partition (rome, which has minimum of 16 cores, I need just 1).</p>
<p>So I switched to snakemake-executor-plugin-slurm using the following config.yaml:</p>
<pre class="lang-yaml prettyprint-override"><code>executor: slurm
default-resources:
  # not necessary if not sure about `slurm_account`, snakemake will try to infer this using `sacctmgr`
  slurm_partition: &quot;rome&quot;
  slurm_extra: &quot;' -o cluster_outputs/smk.{rule}.{jobid}.out -e cluster_outputs/smk.{rule}.{jobid}.err'&quot;
printshellcmds: True
jobs: 100
restart-times: 3
latency-wait: 60
rerun-incomplete: True
use-conda: True
set-threads:
    salmon: 16

set-resources:
    # rule specific resources
  fastqc:
    slurm_partition: staging
</code></pre>
<p>I can send jobs to other partitions! However, these jobs finish and the pipeline never moves past the initial steps as if snakemake never get the &quot;job finished&quot; signal. Also, the jobnames are just integers and the cluster_outputs does not show any logs. When I ctrl-c the main snakemake process I get this error:</p>
<pre class="lang-bash prettyprint-override"><code>[Tue Oct 15 17:13:55 2024]
rule fastqc:
    input: fastq_raw/0053_P2017BB3S19R_S1_R1_001.fastq.gz
    output: qc/fastqc/0053_P2017BB3S19R_S1_R1_001_fastqc.html, qc/fastqc/0053_P2017BB3S19R_S1_R1_001_fastqc.zip
    jobid: 16
    benchmark: benchmark/0053_P2017BB3S19R_S1_R1.fastqc.fastqc.benchmark.tsv
    reason: Missing output files: qc/fastqc/0053_P2017BB3S19R_S1_R1_001_fastqc.zip, qc/fastqc/0053_P2017BB3S19R_S1_R1_001_fastqc.html
    wildcards: sample=0053_P2017BB3S19R_S1, read=R1
    resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=&lt;TBD&gt;, slurm_account=eccdcdc, slurm_partition=staging, slurm_extra= -o cluster_outputs/smk.{rule}.{jobid}.out -e cluster_outputs/smk.{rule}.{jobid}.err


        fastqc fastq_raw/0053_P2017BB3S19R_S1_R1_001.fastq.gz --outdir=qc/fastqc
        
No wall time information given. This might or might not work on your cluster. If not, specify the resource runtime in your rule or as a reasonable default via --default-resources.
Job 16 has been submitted with SLURM jobid sbatch: Single-node jobs run on a shared node by default. Add --exclusive if you want to use a node exclusively.
sbatch: A full node consists of 32 CPU cores, 229376 MiB of memory and 0 GPUs and can be shared by up to 32 jobs.
sbatch: By default shared jobs get 7168 MiB of memory per CPU core, unless explicitly overridden with --mem-per-cpu, --mem-per-gpu or --mem.
sbatch: You will be charged for 1 CPUs, based on the number of CPUs and the amount memory that you've requested.
8177956 (log: /gpfs/home5/me/temp/test_pipeline/.snakemake/slurm_logs/rule_fastqc/0053_P2017BB3S19R_S1_R1/sbatch: Single-node jobs run on a shared node by default. Add --exclusive if you want to use a node exclusively.
sbatch: A full node consists of 32 CPU cores, 229376 MiB of memory and 0 GPUs and can be shared by up to 32 jobs.
sbatch: By default shared jobs get 7168 MiB of memory per CPU core, unless explicitly overridden with --mem-per-cpu, --mem-per-gpu or --mem.
sbatch: You will be charged for 1 CPUs, based on the number of CPUs and the amount memory that you've requested.
8177956.log).
^CTerminating processes on user request, this might take some time.
WorkflowError:
Unable to cancel jobs with scancel (exit code 127): scancel: unrecognized option '--exclusive'
Try &quot;scancel --help&quot; for more information
/bin/sh: line 2: sbatch:: command not found
/bin/sh: line 3: sbatch:: command not found
/bin/sh: line 4: sbatch:: command not found
/bin/sh: line 9: 8177952: command not found
/bin/sh: line 10: sbatch:: command not found
/bin/sh: line 11: sbatch:: command not found
/bin/sh: line 12: sbatch:: command not found
/bin/sh: line 17: 8177954: command not found
/bin/sh: line 18: sbatch:: command not found
/bin/sh: line 19: sbatch:: command not found
/bin/sh: line 20: sbatch:: command not found
/bin/sh: line 25: 8177956: command not found
(snaqs) [me@int4 test_pipeline]$ 
</code></pre>
<p>How can I get best of both worlds? The generic cluster executor works perfectly except for sending job to another the partition, the slurm executor won't finish, most likely I'm missing something.</p>
<p>Thank you!</p>
<p>EDIT:</p>
<p>With the cluster-generic executor I can change the partition succesfully by modifying the sbatch command like this:</p>
<pre class="lang-yaml prettyprint-override"><code>executor: cluster-generic
cluster-generic-submit-cmd:
  sbatch
    --cpus-per-task=16
    --job-name={rule}-{jobid}
    --output=cluster_outputs/{rule}/{rule}-{wildcards}-%j.out
    --parsable
    --partition=staging
</code></pre>
<p>Now the question is, how to specify it per rule...</p>
<p>EDIT:</p>
<p>It seems that when submitting with the slurm executor all jobs have the same name, that could be an issue?</p>
<pre class="lang-bash prettyprint-override"><code>$ squeue -o&quot;%.7i %.9P %.8j %.8u %.2t %.10M %.6D %C %k&quot;
  JOBID PARTITION     NAME     USER ST       TIME  NODES CPUS COMMENT
8178848      rome 9a182ba0 me PD       0:00      1 1 rule_trim_galore_wildcards_0053_P2017BB3S20R_S2
8178845      rome 9a182ba0 me PD       0:00      1 1 rule_trim_galore_wildcards_0053_P2017BB3S19R_S1
8178846   staging 9a182ba0 me CG       0:15      1 1 rule_fastqc_wildcards_0053_P2017BB3S20R_S2_R2
8178847   staging 9a182ba0 me CG       0:15      1 1 rule_fastqc_wildcards_0053_P2017BB3S20R_S2_R1
8178849   staging 9a182ba0 me CG       0:15      1 1 rule_fastqc_wildcards_0053_P2017BB3S19R_S1_R2
8178850   staging 9a182ba0 me CG       0:15      1 1 rule_fastqc_wildcards_0053_P2017BB3S19R_S1_R1
</code></pre>
<p>Oh no of course not, the JOBID is different. I'll leave this here none-the-less.</p>

<snakemake><slurm>
Snakemake on a Slurm cluster
# Snakemake on a Slurm cluster

(Asked by: username_0 on 2024-10-15)

I've been experimenting with Snakemake on a (multi-partition) Slurm cluster (Snellius, the Dutch super computer).
I have switched to Snakemake 8 so that I could make easier use of profiles. So I am using executors. I have tried 2 so far, snakemake-executor-plugin-cluster-generic and snakemake-executor-plugin-slurm.
They both have down-sides, using snakemake-executor-plugin-cluster-generic I can't seem to be able to select the partition that I want, I tried this:
executor: cluster-generic
cluster-generic-submit-cmd:
  sbatch
    --cpus-per-task=16
    --job-name={rule}-{jobid}
    --output=cluster_outputs/{rule}/{rule}-{wildcards}-%j.out
    --parsable
restart-times: 3
max-jobs-per-second: 10
max-status-checks-per-second: 1
local-cores: 1
latency-wait: 60
jobs: 100 # Check what the max is
keep-going: True
rerun-incomplete: True
printshellcmds: True
# scheduler: greedy
use-conda: True

set-threads:
  salmon: 16

set-resources:
  fastqc:
    partition: staging
    # mem_mb: 100

In the submitted job I indeed see: resources: tmpdir=, partition=staging
I also tried:
set-resources:
  fastqc:
    slurm_partition: staging

Again, I see it scrolling by under "resources" however, using squeue and the logs I see that the jobs are still executed on the standard partition (rome, which has minimum of 16 cores, I need just 1).
So I switched to snakemake-executor-plugin-slurm using the following config.yaml:
executor: slurm
default-resources:
  # not necessary if not sure about `slurm_account`, snakemake will try to infer this using `sacctmgr`
  slurm_partition: "rome"
  slurm_extra: "' -o cluster_outputs/smk.{rule}.{jobid}.out -e cluster_outputs/smk.{rule}.{jobid}.err'"
printshellcmds: True
jobs: 100
restart-times: 3
latency-wait: 60
rerun-incomplete: True
use-conda: True
set-threads:
    salmon: 16

set-resources:
    # rule specific resources
  fastqc:
    slurm_partition: staging

I can send jobs to other partitions! However, these jobs finish and the pipeline never moves past the initial steps as if snakemake never get the "job finished" signal. Also, the jobnames are just integers and the cluster_outputs does not show any logs. When I ctrl-c the main snakemake process I get this error:
[Tue Oct 15 17:13:55 2024]
rule fastqc:
    input: fastq_raw/0053_P2017BB3S19R_S1_R1_001.fastq.gz
    output: qc/fastqc/0053_P2017BB3S19R_S1_R1_001_fastqc.html, qc/fastqc/0053_P2017BB3S19R_S1_R1_001_fastqc.zip
    jobid: 16
    benchmark: benchmark/0053_P2017BB3S19R_S1_R1.fastqc.fastqc.benchmark.tsv
    reason: Missing output files: qc/fastqc/0053_P2017BB3S19R_S1_R1_001_fastqc.zip, qc/fastqc/0053_P2017BB3S19R_S1_R1_001_fastqc.html
    wildcards: sample=0053_P2017BB3S19R_S1, read=R1
    resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=, slurm_account=eccdcdc, slurm_partition=staging, slurm_extra= -o cluster_outputs/smk.{rule}.{jobid}.out -e cluster_outputs/smk.{rule}.{jobid}.err


        fastqc fastq_raw/0053_P2017BB3S19R_S1_R1_001.fastq.gz --outdir=qc/fastqc
        
No wall time information given. This might or might not work on your cluster. If not, specify the resource runtime in your rule or as a reasonable default via --default-resources.
Job 16 has been submitted with SLURM jobid sbatch: Single-node jobs run on a shared node by default. Add --exclusive if you want to use a node exclusively.
sbatch: A full node consists of 32 CPU cores, 229376 MiB of memory and 0 GPUs and can be shared by up to 32 jobs.
sbatch: By default shared jobs get 7168 MiB of memory per CPU core, unless explicitly overridden with --mem-per-cpu, --mem-per-gpu or --mem.
sbatch: You will be charged for 1 CPUs, based on the number of CPUs and the amount memory that you've requested.
8177956 (log: /gpfs/home5/me/temp/test_pipeline/.snakemake/slurm_logs/rule_fastqc/0053_P2017BB3S19R_S1_R1/sbatch: Single-node jobs run on a shared node by default. Add --exclusive if you want to use a node exclusively.
sbatch: A full node consists of 32 CPU cores, 229376 MiB of memory and 0 GPUs and can be shared by up to 32 jobs.
sbatch: By default shared jobs get 7168 MiB of memory per CPU core, unless explicitly overridden with --mem-per-cpu, --mem-per-gpu or --mem.
sbatch: You will be charged for 1 CPUs, based on the number of CPUs and the amount memory that you've requested.
8177956.log).
^CTerminating processes on user request, this might take some time.
WorkflowError:
Unable to cancel jobs with scancel (exit code 127): scancel: unrecognized option '--exclusive'
Try "scancel --help" for more information
/bin/sh: line 2: sbatch:: command not found
/bin/sh: line 3: sbatch:: command not found
/bin/sh: line 4: sbatch:: command not found
/bin/sh: line 9: 8177952: command not found
/bin/sh: line 10: sbatch:: command not found
/bin/sh: line 11: sbatch:: command not found
/bin/sh: line 12: sbatch:: command not found
/bin/sh: line 17: 8177954: command not found
/bin/sh: line 18: sbatch:: command not found
/bin/sh: line 19: sbatch:: command not found
/bin/sh: line 20: sbatch:: command not found
/bin/sh: line 25: 8177956: command not found
(snaqs) [me@int4 test_pipeline]$ 

How can I get best of both worlds? The generic cluster executor works perfectly except for sending job to another the partition, the slurm executor won't finish, most likely I'm missing something.
Thank you!
EDIT:
With the cluster-generic executor I can change the partition succesfully by modifying the sbatch command like this:
executor: cluster-generic
cluster-generic-submit-cmd:
  sbatch
    --cpus-per-task=16
    --job-name={rule}-{jobid}
    --output=cluster_outputs/{rule}/{rule}-{wildcards}-%j.out
    --parsable
    --partition=staging

Now the question is, how to specify it per rule...
EDIT:
It seems that when submitting with the slurm executor all jobs have the same name, that could be an issue?
$ squeue -o"%.7i %.9P %.8j %.8u %.2t %.10M %.6D %C %k"
  JOBID PARTITION     NAME     USER ST       TIME  NODES CPUS COMMENT
8178848      rome 9a182ba0 me PD       0:00      1 1 rule_trim_galore_wildcards_0053_P2017BB3S20R_S2
8178845      rome 9a182ba0 me PD       0:00      1 1 rule_trim_galore_wildcards_0053_P2017BB3S19R_S1
8178846   staging 9a182ba0 me CG       0:15      1 1 rule_fastqc_wildcards_0053_P2017BB3S20R_S2_R2
8178847   staging 9a182ba0 me CG       0:15      1 1 rule_fastqc_wildcards_0053_P2017BB3S20R_S2_R1
8178849   staging 9a182ba0 me CG       0:15      1 1 rule_fastqc_wildcards_0053_P2017BB3S19R_S1_R2
8178850   staging 9a182ba0 me CG       0:15      1 1 rule_fastqc_wildcards_0053_P2017BB3S19R_S1_R1

Oh no of course not, the JOBID is different. I'll leave this here none-the-less.

**Tags:** snakemake, slurm

---

## 1 answers

### Answer by username_0 on 2024-10-27. Score: 0 | Accepted answer

Ok, I think the Slurm Executor is the preferred way, but there is an issue wrt my cluster (the last version does not work, but 3 version back does), I have filed an issue on GitHub: https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/157, hope to resolve this soon, I'll put my efforts towards getting the Slurm executor working well for me.
FYI, it turns out that in my case I probably shouldn't send jobs to other partitions in my particular case.
Moreover, this code from above was a mistake, the Slurm executor does not accept this:
-o cluster_outputs/smk.{rule}.{jobid}.out -e cluster_outputs/smk.{rule}.{jobid}.err'

And probably for the better because once you get started with job-grouping that line does not make sense anymore (there are now multiple rules or samples per job!).


---

