<p>Got here from your slurm-excecutor slurm issue :)</p>
<p>First thing's first, my use of snakemake is not for bioinformatics, I use it in a completely different field where we dont need that many small parallel jobs. I do use job grouping and to my understanding it should work as you intend it to.</p>
<p>Also a disclaimer, I use snakemake 7.X. I still have not upgraded my workflow to work with 8.X, hence, I use the &quot;old&quot; slurm executor (the cookiecutter profile), but I hope my experience translates to 8.X if that's what you are working with.</p>
<p>Now to your questions:
The log file for a grouped rule should group all the small jobs submitted with the group. So in your case check:</p>
<blockquote>
<p>log: /gpfs/scratch1/shared/fvhemert/temp/tests/.snakemake/slurm_logs/rule_fastqc/0053_P2017BB3S19R_S1_R2/8281752.log</p>
</blockquote>
<p>If all the grouped rules are show in the log, then success. And yes, each group should only spawn one job submission with a unique jobID.
It is kind of strange that the folder where the log is saved says &quot;rule&quot; in it, in my case logs are saved in a folder with the name of the group. This might be a difference between the profile and the executor.</p>
<p>Something else that might be useful. After submitting the job, just spam squeue and see how your job queue looks like. Try with and without grouping, the difference should be quite clear.</p>
<p>Creating groups with a wildcard in it is something I never tried, so can't help you with that one.</p>
<p>Resource wise, remember you are being charged 16 cores anyways, which means you also get the memory allocation for that (By default 1.75gb per thread in a thin node). So if you have 4 rules that run in parallel and each of them needs... say 6gb, you can just specify the rule needs 4 threads and by default it will get 1.75*4gb of RAM, and since the 4 rules &quot;fit&quot; in the allocated node, they should run in parallel.
In other words, I stopped worrying about optimizing for memory, and I now &quot;optimize&quot; by threads (even though I know many of my rules use just 1 core), and fine-tune from there!</p>

CC BY-SA 4.0
2024-11-08T19:43:05.900Z
Ok I got it to work, but only by defining goups in the rules, it doesn't work from the config yaml. I think I got the concept wrong, it seems that it only groups subsequent jobs? Can it also be used to make for example 16 small jobs be run each in 1 thread on a 16 core CPU?
Freek
CC BY-SA 4.0
2024-11-08T20:41:10.503Z
Ok, I think I have to keep in mind that you can only group tasks that are connected in the DAG, right?
Freek
CC BY-SA 4.0
2024-11-08T21:23:13.913Z
Ok,I'm running into this issue, will update here if I get it to work: https://github.com/snakemake/snakemake/issues/3165
Freek
CC BY-SA 4.0
2024-10-30T16:13:25.533Z
2024-10-30T16:19:53.550Z
2024-10-30T16:19:53.550Z
I Farias
<p>According to this answer to an issue, the group definition should not be part of the cluster configuration:</p>
<p><a href="https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/171#issuecomment-2466746165" rel="nofollow noreferrer">https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/171#issuecomment-2466746165</a></p>
<p>(Edit: I just realized that this was probably an issue you created)</p>
<p>Groups should be defined in the snakefile, in the workflow config, or on the command-line.</p>
<p>However, ensure you use a recent enough version of snakemake if you try to define groups in the workflow profile, because there was a bug that prevented this to be taken into account: <a href="https://github.com/snakemake/snakemake/issues/3039" rel="nofollow noreferrer">https://github.com/snakemake/snakemake/issues/3039</a></p>
<p>This was fixed in version 8.22 (released last month): <a href="https://github.com/snakemake/snakemake/compare/v8.21.0...v8.22.0" rel="nofollow noreferrer">https://github.com/snakemake/snakemake/compare/v8.21.0...v8.22.0</a></p>

CC BY-SA 4.0
2024-11-22T12:49:46.403Z
2024-11-22T12:49:46.403Z
bli
<p>I have been trying to use Snakemake on the Dutch National Super Computer, Snellius, which is just a large Slurm cluster.</p>
<p>On Snellius the minimum amount of cores you can use is 16 per node. So in order to make efficient use of its resources I want to use Snakemake Job Grouping. So far I have not been able to get it to work, but I admit I also don't have enough knowledge of slurm to determine if everything worked as planned...</p>
<p>If I understand the documentation right (but I don't feel I do), I can just assign rules to a group, and they will be executed on 1 node, together. I also understand that you cannot randomly group stuff when they are far apart in the DAG. I tried grouping for fastqc using the following cluster configuration file:</p>
<pre class="lang-yaml prettyprint-override"><code>executor: slurm
default-resources:
  slurm_partition: &quot;rome&quot; # Default partition
  mem_mb: 1000 # By default jobs get 1000 MB of memory
  runtime: 60 # All tasks killed by default after 1 hour
  slurm_extra: &quot;'-o cluster_outputs/%j.out -e cluster_outputs/%j.err'&quot;
printshellcmds: True
jobs: 100
restart-times: 3 # Re-try failed jobs 3 times, especially handy for SRA downloads which may fail randomly
latency-wait: 60 # Wait for 60 secs before deciding a file is missing (perhaps there is filesystem latency)
keep-going: True # Even if one process fails, keep going with the rest of the independent tasks
rerun-incomplete: True # If a task didn't finish, re-run on the next restart
use-conda: True # Build all env automatically
conda-prefix: /home/fvhemert/projects/snaqs_files/snakemake_envs # Store the envs centrally
group-components: fastqc_group=16 # Not sure what this means

# rule specific resources
set-resources:
  download_sra_files:
    threads: 1
    runtime: 120
  sra2fastq:
    threads: 6
    runtime: 360
  fastqc:
    threads: 1
    group: fastqc_group
  trim_galore:
    threads: 4
    runtime: 720
    mem_mb: 20000
  complexity_20mer_counter:
    mem_mb: 10000
  salmon:
    threads: 16
    mem_mb: 20000 # This seems to work, but can we do with less?
    runtime: 600
    group: &quot;'group_{sample}'&quot;
  tpm4_normalization:
    mem_mb: 10000 # How much do we really need?
    runtime: 30
    group: &quot;'group_{sample}'&quot;
  customqc_parameters:
    mem_mb: 10000 # How much do we really need?
    runtime: 30
    group: &quot;'group_{sample}'&quot;
</code></pre>
<p>In the snakefile I have added this line (but I wonder if this is necessary!):</p>
<pre class="lang-py prettyprint-override"><code>groups = [&quot;fastqc_group&quot;, &quot;group_{sample}&quot;]
</code></pre>
<p>I think there is another use of Job Grouping, where can group all tasks per sample so that I can make sure that they run on the same node and can access the same local fast SSD storage. That is what I have tried to implement with the group_{sample}.</p>
<p>If I understand correctly, the way to check if job grouping is actually working is to do a dry run and see if you actually create less jobs, for example in the overview below there should be 1 fastqc job to be submitted (which would then run all 4 in parallel). This has never happened for me, the dry run always gives this overview:</p>
<pre class="lang-bash prettyprint-override"><code>...
job                           count
--------------------------  -------
all                               1
complexity_20mer_counter          4
create_flagged_sampletable        1
create_pcs_raw_files              2
customqc_parameters               2
customqc_report                   1
fastqc                            4
qc_flagging                       2
rnaseq_multiqc                    1
salmon                            2
seqrun_expression_reports         1
tpm4_normalization                2
trim_galore                       2
total                            25

Reasons:
    (check individual jobs above for details)
    input files updated by another job:
        all, complexity_20mer_counter, create_flagged_sampletable, create_pcs_raw_files, customqc_parameters, customqc_report, qc_flagging, rnaseq_multiqc, salmon, seqrun_expression_reports, tpm4_normalization
    output files have to be generated:
        trim_galore
    updated input files:
        fastqc, trim_galore

This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.
</code></pre>
<p>Starting the workflow you can see that the jobs get a &quot;group=fastqc_group&quot; parameter:</p>
<pre class="lang-bash prettyprint-override"><code>[Sun Oct 27 22:45:27 2024]
rule fastqc:
    input: fastq_raw/0053_P2017BB3S19R_S1_R2_001.fastq.gz
    output: qc/fastqc/0053_P2017BB3S19R_S1_R2_001_fastqc.html, qc/fastqc/0053_P2017BB3S19R_S1_R2_001_fastqc.zip
    jobid: 17
    benchmark: benchmark/0053_P2017BB3S19R_S1_R2.fastqc.fastqc.benchmark.tsv
    reason: Missing output files: qc/fastqc/0053_P2017BB3S19R_S1_R2_001_fastqc.html, qc/fastqc/0053_P2017BB3S19R_S1_R2_001_fastqc.zip
    wildcards: sample=0053_P2017BB3S19R_S1, read=R2
    resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=&lt;TBD&gt;, slurm_partition=rome, runtime=60, slurm_extra=-o cluster_outputs/%j.out -e cluster_outputs/%j.err, threads=1, group=fastqc_group


        fastqc fastq_raw/0053_P2017BB3S19R_S1_R2_001.fastq.gz --outdir=qc/fastqc
        
Job 17 has been submitted with SLURM jobid 8281752 (log: /gpfs/scratch1/shared/fvhemert/temp/tests/.snakemake/slurm_logs/rule_fastqc/0053_P2017BB3S19R_S1_R2/8281752.log).
</code></pre>
<p>The workflow finishes successfully but there is no sign of any job grouping... But perhaps I'm just missing it? I also have no idea what it should look like, perhaps the cluster output files contain the logs of all the grouped jobs, all grouped jobs should have a single JobID, right?? Something like that? Any pointers are welcome.</p>
<p>If anyone could point me to some functional examples next to the documentation that would be very handy. If I get this to work I will certainly post my workflow + configs somewhere as an example.</p>

<snakemake><slurm>
Snakemake on Slurm and job grouping
# Snakemake on Slurm and job grouping

(Asked by: username_0 on 2024-10-27)

I have been trying to use Snakemake on the Dutch National Super Computer, Snellius, which is just a large Slurm cluster.
On Snellius the minimum amount of cores you can use is 16 per node. So in order to make efficient use of its resources I want to use Snakemake Job Grouping. So far I have not been able to get it to work, but I admit I also don't have enough knowledge of slurm to determine if everything worked as planned...
If I understand the documentation right (but I don't feel I do), I can just assign rules to a group, and they will be executed on 1 node, together. I also understand that you cannot randomly group stuff when they are far apart in the DAG. I tried grouping for fastqc using the following cluster configuration file:
executor: slurm
default-resources:
  slurm_partition: "rome" # Default partition
  mem_mb: 1000 # By default jobs get 1000 MB of memory
  runtime: 60 # All tasks killed by default after 1 hour
  slurm_extra: "'-o cluster_outputs/%j.out -e cluster_outputs/%j.err'"
printshellcmds: True
jobs: 100
restart-times: 3 # Re-try failed jobs 3 times, especially handy for SRA downloads which may fail randomly
latency-wait: 60 # Wait for 60 secs before deciding a file is missing (perhaps there is filesystem latency)
keep-going: True # Even if one process fails, keep going with the rest of the independent tasks
rerun-incomplete: True # If a task didn't finish, re-run on the next restart
use-conda: True # Build all env automatically
conda-prefix: /home/fvhemert/projects/snaqs_files/snakemake_envs # Store the envs centrally
group-components: fastqc_group=16 # Not sure what this means

# rule specific resources
set-resources:
  download_sra_files:
    threads: 1
    runtime: 120
  sra2fastq:
    threads: 6
    runtime: 360
  fastqc:
    threads: 1
    group: fastqc_group
  trim_galore:
    threads: 4
    runtime: 720
    mem_mb: 20000
  complexity_20mer_counter:
    mem_mb: 10000
  salmon:
    threads: 16
    mem_mb: 20000 # This seems to work, but can we do with less?
    runtime: 600
    group: "'group_{sample}'"
  tpm4_normalization:
    mem_mb: 10000 # How much do we really need?
    runtime: 30
    group: "'group_{sample}'"
  customqc_parameters:
    mem_mb: 10000 # How much do we really need?
    runtime: 30
    group: "'group_{sample}'"

In the snakefile I have added this line (but I wonder if this is necessary!):
groups = ["fastqc_group", "group_{sample}"]

I think there is another use of Job Grouping, where can group all tasks per sample so that I can make sure that they run on the same node and can access the same local fast SSD storage. That is what I have tried to implement with the group_{sample}.
If I understand correctly, the way to check if job grouping is actually working is to do a dry run and see if you actually create less jobs, for example in the overview below there should be 1 fastqc job to be submitted (which would then run all 4 in parallel). This has never happened for me, the dry run always gives this overview:
...
job                           count
--------------------------  -------
all                               1
complexity_20mer_counter          4
create_flagged_sampletable        1
create_pcs_raw_files              2
customqc_parameters               2
customqc_report                   1
fastqc                            4
qc_flagging                       2
rnaseq_multiqc                    1
salmon                            2
seqrun_expression_reports         1
tpm4_normalization                2
trim_galore                       2
total                            25

Reasons:
    (check individual jobs above for details)
    input files updated by another job:
        all, complexity_20mer_counter, create_flagged_sampletable, create_pcs_raw_files, customqc_parameters, customqc_report, qc_flagging, rnaseq_multiqc, salmon, seqrun_expression_reports, tpm4_normalization
    output files have to be generated:
        trim_galore
    updated input files:
        fastqc, trim_galore

This was a dry-run (flag -n). The order of jobs does not reflect the order of execution.

Starting the workflow you can see that the jobs get a "group=fastqc_group" parameter:
[Sun Oct 27 22:45:27 2024]
rule fastqc:
    input: fastq_raw/0053_P2017BB3S19R_S1_R2_001.fastq.gz
    output: qc/fastqc/0053_P2017BB3S19R_S1_R2_001_fastqc.html, qc/fastqc/0053_P2017BB3S19R_S1_R2_001_fastqc.zip
    jobid: 17
    benchmark: benchmark/0053_P2017BB3S19R_S1_R2.fastqc.fastqc.benchmark.tsv
    reason: Missing output files: qc/fastqc/0053_P2017BB3S19R_S1_R2_001_fastqc.html, qc/fastqc/0053_P2017BB3S19R_S1_R2_001_fastqc.zip
    wildcards: sample=0053_P2017BB3S19R_S1, read=R2
    resources: mem_mb=1000, mem_mib=954, disk_mb=1000, disk_mib=954, tmpdir=, slurm_partition=rome, runtime=60, slurm_extra=-o cluster_outputs/%j.out -e cluster_outputs/%j.err, threads=1, group=fastqc_group


        fastqc fastq_raw/0053_P2017BB3S19R_S1_R2_001.fastq.gz --outdir=qc/fastqc
        
Job 17 has been submitted with SLURM jobid 8281752 (log: /gpfs/scratch1/shared/fvhemert/temp/tests/.snakemake/slurm_logs/rule_fastqc/0053_P2017BB3S19R_S1_R2/8281752.log).

The workflow finishes successfully but there is no sign of any job grouping... But perhaps I'm just missing it? I also have no idea what it should look like, perhaps the cluster output files contain the logs of all the grouped jobs, all grouped jobs should have a single JobID, right?? Something like that? Any pointers are welcome.
If anyone could point me to some functional examples next to the documentation that would be very handy. If I get this to work I will certainly post my workflow + configs somewhere as an example.

**Tags:** snakemake, slurm

---

## 2 answers

### Answer by username_2 on 2024-11-22. Score: 2 | Accepted answer

According to this answer to an issue, the group definition should not be part of the cluster configuration:

<https://github.com/snakemake/snakemake-executor-plugin-slurm/issues/171#issuecomment-2466746165>

(Edit: I just realized that this was probably an issue you created)

Groups should be defined in the snakefile, in the workflow config, or on the command-line.

However, ensure you use a recent enough version of snakemake if you try to define groups in the workflow profile, because there was a bug that prevented this to be taken into account: <https://github.com/snakemake/snakemake/issues/3039>

This was fixed in version 8.22 (released last month): <https://github.com/snakemake/snakemake/compare/v8.21.0...v8.22.0>


---

### Answer by username_1 on 2024-10-30. Score: 2

Got here from your slurm-excecutor slurm issue :)

First thing's first, my use of snakemake is not for bioinformatics, I use it in a completely different field where we dont need that many small parallel jobs. I do use job grouping and to my understanding it should work as you intend it to.

Also a disclaimer, I use snakemake 7.X. I still have not upgraded my workflow to work with 8.X, hence, I use the "old" slurm executor (the cookiecutter profile), but I hope my experience translates to 8.X if that's what you are working with.

Now to your questions:
The log file for a grouped rule should group all the small jobs submitted with the group. So in your case check:

> log: /gpfs/scratch1/shared/fvhemert/temp/tests/.snakemake/slurm\_logs/rule\_fastqc/0053\_P2017BB3S19R\_S1\_R2/8281752.log

If all the grouped rules are show in the log, then success. And yes, each group should only spawn one job submission with a unique jobID.
It is kind of strange that the folder where the log is saved says "rule" in it, in my case logs are saved in a folder with the name of the group. This might be a difference between the profile and the executor.

Something else that might be useful. After submitting the job, just spam squeue and see how your job queue looks like. Try with and without grouping, the difference should be quite clear.

Creating groups with a wildcard in it is something I never tried, so can't help you with that one.

Resource wise, remember you are being charged 16 cores anyways, which means you also get the memory allocation for that (By default 1.75gb per thread in a thin node). So if you have 4 rules that run in parallel and each of them needs... say 6gb, you can just specify the rule needs 4 threads and by default it will get 1.75\*4gb of RAM, and since the 4 rules "fit" in the allocated node, they should run in parallel.
In other words, I stopped worrying about optimizing for memory, and I now "optimize" by threads (even though I know many of my rules use just 1 core), and fine-tune from there!

> **Comments:**
> **username_0**: Ok I got it to work, but only by defining goups in the rules, it doesn't work from the config yaml. I think I got the concept wrong, it seems that it only groups subsequent jobs? Can it also be used to make for example 16 small jobs be run each in 1 thread on a 16 core CPU?  
> **username_0**: Ok, I think I have to keep in mind that you can only group tasks that are connected in the DAG, right?  
> **username_0**: Ok,I'm running into this issue, will update here if I get it to work: https://github.com/snakemake/snakemake/issues/3165  

---

