<p>It looks like you can just bundle your cluster config in its own YAML file, and reference that from within a profile YAML.  Here's a clumsy little example using a fake job submission script: <a href="https://github.com/ressy/example-snakemake-profiles" rel="nofollow noreferrer">https://github.com/ressy/example-snakemake-profiles</a></p>

<hr>

<p>The <a href="https://snakemake.readthedocs.io/en/stable/snakefiles/configuration.html#cluster-configuration-deprecated" rel="nofollow noreferrer">current documentation</a> makes it sound like cluster configuration support is going away and being replaced by profiles:</p>

<blockquote>
  <p>While still being possible, cluster configuration has been deprecated by the introduction of Profiles.</p>
</blockquote>

<p>...but from those example profiles, that might be misleading; here's an example template for a profile YAML that references a separate cluster-config YAML:</p>

<p><a href="https://github.com/Snakemake-Profiles/generic/blob/master/%7B%7Bcookiecutter.profile_name%7D%7D/config.yaml" rel="nofollow noreferrer">https://github.com/Snakemake-Profiles/generic/blob/master/%7B%7Bcookiecutter.profile_name%7D%7D/config.yaml</a></p>

<pre><code>...
cluster-config: "$((INSTALDIR))/overwrite_cluster_param.yaml" #abs path
...
</code></pre>

<p>That cluster-config YAML has the same default and per-rule entries just your JSON version does:</p>

<p><a href="https://github.com/Snakemake-Profiles/generic/blob/master/%7B%7Bcookiecutter.profile_name%7D%7D/overwrite_cluster_param.yaml" rel="nofollow noreferrer">https://github.com/Snakemake-Profiles/generic/blob/master/%7B%7Bcookiecutter.profile_name%7D%7D/overwrite_cluster_param.yaml</a></p>

<pre><code>...
# specific parameters for certain rules, which need more time/memory

#run_assembler:
#  queue: bigmem
#  time: 100
#   threads and memory definen in config file
</code></pre>

<p>So I think you can just set up a pair of YAML files accordingly including options for your rsem_quantify and star_map_human rules.  From my own test it looks like it handles the cluster argument templating just as it did before when using a profile.  (If this is correct we should nag them to make the new documentation more explicit though.)</p>

CC BY-SA 4.0
2020-02-19T19:38:12.100Z
2020-02-19T19:38:12.100Z
Jesse
<p>I am an avid user of Snakemake. Recently we have been refreshing our pipelines and I saw that a cluster.json file is no longer the recommended way to store the cluster configuration.</p>

<p>I used to start my pipelines like this:</p>

<pre><code>snakemake --cluster-config cluster.json --cluster "qsub -W umask=002 -d cluster_outputs  -l nodes={cluster.nodes}:ppn={cluster.ppn} -N {rule}" --jobs 245 --use-conda --rerun-incomplete --keep-going --latency-wait 60
</code></pre>

<p>With the cluster.json looking like this:</p>

<pre><code>{
    "__default__" :
    {
        "cluster"   : "qsub",
        "jobs"      : 70 ,
        "nodes"     : 1,
        "ppn"       : 1
    },
    "rsem_quantify" :
    {
        "ppn"       : 8
    },
    "star_map_human" :
    {
        "ppn"       : 4
    },
}
</code></pre>

<p>Now, I understand that some of this goes in to the new profiles.yaml like this:</p>

<pre><code>cluster: qsub
jobs: 70
</code></pre>

<p>But how do I set the ppn values per rule? And how do I set the other configuration options into profile.yaml (like: snakemake "qsub -W umask=002 -d cluster_outputs  -l nodes={cluster.nodes}:ppn={cluster.ppn} -N {rule}", etc.)</p>

<p>Any examples are appreciated, <a href="https://github.com/snakemake-profiles" rel="noreferrer">https://github.com/snakemake-profiles</a> is not really helping</p>

CC BY-SA 4.0
2020-03-05T19:15:04.130Z
Just wanted to note that I asked the [same question in stackoverflow](https://stackoverflow.com/q/60311167/3998252). No  good answer yet though.
Manavalan Gajapathy
<snakemake>
Snakemake: Migrating from deprecated cluster.json to new profiles.yaml
# Snakemake: Migrating from deprecated cluster.json to new profiles.yaml

(Asked by: username_0 on 2020-02-19)

I am an avid user of Snakemake. Recently we have been refreshing our pipelines and I saw that a cluster.json file is no longer the recommended way to store the cluster configuration.
I used to start my pipelines like this:
snakemake --cluster-config cluster.json --cluster "qsub -W umask=002 -d cluster_outputs  -l nodes={cluster.nodes}:ppn={cluster.ppn} -N {rule}" --jobs 245 --use-conda --rerun-incomplete --keep-going --latency-wait 60

With the cluster.json looking like this:
{
    "__default__" :
    {
        "cluster"   : "qsub",
        "jobs"      : 70 ,
        "nodes"     : 1,
        "ppn"       : 1
    },
    "rsem_quantify" :
    {
        "ppn"       : 8
    },
    "star_map_human" :
    {
        "ppn"       : 4
    },
}

Now, I understand that some of this goes in to the new profiles.yaml like this:
cluster: qsub
jobs: 70

But how do I set the ppn values per rule? And how do I set the other configuration options into profile.yaml (like: snakemake "qsub -W umask=002 -d cluster_outputs  -l nodes={cluster.nodes}:ppn={cluster.ppn} -N {rule}", etc.)
Any examples are appreciated, https://github.com/snakemake-profiles is not really helping

**Tags:** snakemake
> **Comments:**
> **username_1**: Just wanted to note that I asked the [same question in stackoverflow](https://stackoverflow.com/q/60311167/3998252). No  good answer yet though.  

---

## 1 answers

### Answer by username_2 on 2020-02-19. Score: 1

It looks like you can just bundle your cluster config in its own YAML file, and reference that from within a profile YAML.  Here's a clumsy little example using a fake job submission script: https://github.com/ressy/example-snakemake-profiles

The current documentation makes it sound like cluster configuration support is going away and being replaced by profiles:

While still being possible, cluster configuration has been deprecated by the introduction of Profiles.

...but from those example profiles, that might be misleading; here's an example template for a profile YAML that references a separate cluster-config YAML:
https://github.com/Snakemake-Profiles/generic/blob/master/%7B%7Bcookiecutter.profile_name%7D%7D/config.yaml
...
cluster-config: "$((INSTALDIR))/overwrite_cluster_param.yaml" #abs path
...

That cluster-config YAML has the same default and per-rule entries just your JSON version does:
https://github.com/Snakemake-Profiles/generic/blob/master/%7B%7Bcookiecutter.profile_name%7D%7D/overwrite_cluster_param.yaml
...
# specific parameters for certain rules, which need more time/memory

#run_assembler:
#  queue: bigmem
#  time: 100
#   threads and memory definen in config file

So I think you can just set up a pair of YAML files accordingly including options for your rsem_quantify and star_map_human rules.  From my own test it looks like it handles the cluster argument templating just as it did before when using a profile.  (If this is correct we should nag them to make the new documentation more explicit though.)


---

