<p><strong>DRMAA</strong> (Distributed Resource Management Application API) appears to be an open API that describes a specification for submission and management of work submitted to some grid/cluster. <a href="https://en.wikipedia.org/wiki/DRMAA#Implementations" rel="nofollow noreferrer">If your scheduler is DRMAA compliant</a>, my guess would be that using <code>snakemake</code>'s <code>--drmaa</code> flag will afford you the additional controls exposed by that API.</p>

<p>As mentioned in your question with DRMAA support, a Ctrl+C on one's console would be passed on to the grid to kill or stop submitted jobs. A task that would otherwise require use of the <code>qdel</code> command on <code>Sun Grid Engine</code>, for example.</p>

<p>I could see why the suggestion would be to enable <code>--drmaa</code> where possible as it's likely to expose functionality (such as control of jobs, and availability of logs and errors) that makes the submission and management of jobs a little easier for you and <code>snakemake</code>. Perhaps someone with <code>snakemake</code> experience can explain a little more, but I hope this helps?</p>

<p>For what it's worth, I use Sun Grid Engine without DRMAA and it's quite a pain to get good information out of <code>qstat</code>, and its <code>qacct</code> log format is quite possibly the worst file format I have ever encountered. I suspect DRMAA provides a nice (or at least more reasonable) API for <code>snakemake</code> (and others) to get at this information more readily.</p>

<p><strong>Appended:</strong></p>

<p>Under the hood, the flags are parsed by Python's <code>argparse</code> module in the <code>__init__.py</code>. <a href="https://bitbucket.org/snakemake/snakemake/src/e7bdc7587c6e41113b972e3eba1b9404e741873c/snakemake/__init__.py?fileviewer=file-view-default#__init__.py-747" rel="nofollow noreferrer">A mutually exclusive group of options</a> forces a selection of <code>--cluster</code> or <code>--drmaa</code> (or <code>--cluster-sync</code>).</p>

<p><code>--cluster</code> or <code>-c</code> requires you to specify an argument; the command for which to submit a job on your cluster (the example names <code>qsub</code>). The <code>--drmaa</code> flag just seems to indicate to <code>snakemake</code> that DRMAA is to be used, which doesn't change a lot in terms of how the command you run from your console is processed in the <code>__init__.py</code> or <a href="https://bitbucket.org/snakemake/snakemake/src/e7bdc7587c6e41113b972e3eba1b9404e741873c/snakemake/workflow.py?at=master&amp;fileviewer=file-view-default" rel="nofollow noreferrer"><code>workflow.py</code></a>.</p>

<p>However, when it is time to interface with your scheduler (as specified in <a href="https://bitbucket.org/snakemake/snakemake/src/e7bdc7587c6e41113b972e3eba1b9404e741873c/snakemake/scheduler.py?at=master&amp;fileviewer=file-view-default" rel="nofollow noreferrer"><code>scheduler.py</code></a>), an <code>elif</code> statement checks whether you're in some form of cluster mode and the <code>else</code> catches the case where you've raised the <code>--drmaa</code> flag instead.</p>

<p>Here's where the magic happens, as now your job is submitted with the <code>DRMAAExecutor</code> as specified in <a href="https://bitbucket.org/snakemake/snakemake/src/e7bdc7587c6e41113b972e3eba1b9404e741873c/snakemake/executors.py?at=master&amp;fileviewer=file-view-default" rel="nofollow noreferrer">executors.py</a>, instead of the default executor.</p>

<p>Without investigating much more, I can see that the <code>DRMAAExecutor</code> features some more class attributes and exposes additional functions when compared to the others.</p>

<p>Hope this is of more use? Again, I've only just looked at the code myself but in lieu of a <code>snakemake</code> user chiming in, I thought I'd have a nose.</p>

CC BY-SA 3.0
2017-06-06T15:36:18.903Z
Thanks, but this tells me what I already know; i have a conceptual understanding but i'd like the details on how the 2 different approaches are implemented in snakemake, i'll try to clarify the question a bit
Chris_Rands
CC BY-SA 3.0
2017-06-06T15:37:21.883Z
Sorry I pre-empted what you might have been asking for as I didn't have much more room to write in the comments. I'll try and expand to address your edited question if I can help!
Sam Nicholls
CC BY-SA 3.0
2017-06-06T16:05:59.697Z
@Chris_Rands I've expanded, is this relevant?
Sam Nicholls
CC BY-SA 3.0
2017-06-06T20:31:12.203Z
Thanks, upvoted, sorry if I wasn't very clear with my question, I just thought this might be useful question for others,i was planning on answering myself if i didn't get any answers, but i've not delved into the source yet
Chris_Rands
CC BY-SA 3.0
2017-06-06T20:33:04.943Z
@Chris_Rands No problem (and thanks)! I enjoy digging around code bases to see how they work. Between Devon and I, have we found what you were looking for? Happy to chat on this with you if not.
Sam Nicholls
CC BY-SA 3.0
2017-06-06T15:31:11.623Z
2017-06-06T18:54:34.823Z
2017-06-06T18:54:34.823Z
Sam Nicholls
<p>I'd always kind of wondered how this worked too, so I took this as an excuse to look into the snakemake code. At the end of the day this becomes a question of (1) how are jobs actually submitted and (2) how is it determined if jobs are done (and then whether they failed)?</p>

<p>For DRMAA, python has a module (appropriately named "drmaa") that wraps around the libdrmaa library that comes with most schedulers. This is a very popular route, for example the Galaxy project uses this for dealing with most clusters (e.g., I use it to connect our internal Galaxy instance to our slurm cluster). The huge benefit here is that DRMAA does magic to allow you to simply submit commands to your cluster without needing to know whether you should execute <code>qsub</code> or <code>srun</code> or something else. Further, it then provides methods to simply query whether a job is running or not and what its exit status was.</p>

<p>Using the <code>--cluster</code> command requires a LOT more magic on the side of snakemake. At the end of the day it creates a shell script that it then submits using the command you provided. Importantly, it includes some secret files in that script that it can then watch for (ever noticed that <code>.snakemake</code> directory where you execute a script? That appears to be among the things it's used for.). These are named <code>{jobid}.jobfinished</code> and <code>{jobid}.jobfailed</code> and one of them will get touched, depending on the exit status of your command/script. Once one of those is there, then snakemake can move on in its DAG (or not, if there's a failure). This is obviously a LOT more to keep track of and it's then not possible for snakemake to cancel running jobs, which is something one can do easily with DRMAA.</p>

CC BY-SA 3.0
2017-06-06T19:42:57.710Z
Having written a [package for submitting jobs to Sun Grid Engine](https://github.com/SamStudio8/sunblock) without DRMAA, can definitely confirm the magic must be strong here. Though instead of monitoring whether files have been touched, I chose to fire a POST request at the monitoring server. I can see that DRMAA must expose some nice API to get job status without having to do some odd workarounds.
Sam Nicholls
CC BY-SA 3.0
2017-06-06T19:50:11.410Z
Yup, it's more convenient than what I would have come up with too :)
Devon Ryan
CC BY-SA 3.0
2017-06-06T19:38:15.237Z
2017-06-06T19:38:15.237Z
Devon Ryan
<p>I'm fairly new to <a href="https://snakemake.readthedocs.io/en/stable/" rel="noreferrer">snakemake</a> and I'm trying to understand the difference between the <code>--cluster</code> and <code>--drmaa</code> flags, both of which are used to submit jobs to compute clusters/nodes.</p>

<p>The docs give a few hints about the advantages of using <code>--drmaa</code> <a href="http://snakemake.readthedocs.io/en/stable/executable.html" rel="noreferrer">here</a>:</p>

<blockquote>
  <p>If your cluster system supports DRMAA, Snakemake can make use of that
  to increase the control over jobs. E.g. jobs can be cancelled upon
  pressing Ctrl+C, which is not possible with the generic --cluster
  support.</p>
</blockquote>

<p>And <a href="http://snakemake.readthedocs.io/en/stable/tutorial/additional_features.html" rel="noreferrer">here</a>:</p>

<blockquote>
  <p>If available, DRMAA is preferable over the generic cluster modes
  because it provides better control and error handling.</p>
</blockquote>

<p>So I have a conceptual understanding of the advantages of using <code>--drmaa</code>. However, I don't consider the above a very complete explanation, and I don't know how these flags are implemented in snakemake under the hood, can someone elaborate?</p>

<p>Note, that although this could be considered a more general programming question, snakemake is predominantly used in bioinformatics; I was persuaded that this question would be considered on-topic because of <a href="https://bioinformatics.meta.stackexchange.com/questions/61/serving-as-a-support-forum-for-specific-tools">this meta-post</a> and <a href="https://bioinformatics.meta.stackexchange.com/a/66/104">this answer</a>. </p>

CC BY-SA 3.0
2017-06-06T15:18:29.757Z
Hi Chris! What's the actual question here? Seems that DRMAA is an open API describing submission and control of cluster/grid jobs. So if you have a cluster or submission system that has support for it (such as SLURM), `snakemake` will afford you the additional controls as supported by that API -- including as per your question, the ability to pass on a Ctrl+C at your console to the grid -- a scenario that would require use of the `qdel` command on Sun Grid Engine, for example. So if you've infrastructure with a scheduler that has DRMAA support, I can see why you might as well use `--drmaa`.
Sam Nicholls
CC BY-SA 3.0
2017-06-06T19:01:07.707Z
Maybe wait until we've actually poached the snakemake developers :)
Devon Ryan
CC BY-SA 3.0
2017-06-06T19:26:24.387Z
The code is open source and fairly easy to follow, I found it quite easy to work out how DRMAA jobs were submitted by nosing around the source for 10 minutes. I'm not entirely sure what the question is looking for, though.
Sam Nicholls
<snakemake>
How are snakemake's --cluster and --drmaa options implemented?
# How are snakemake's --cluster and --drmaa options implemented?

(Asked by: username_0 on 2017-06-06)

I'm fairly new to [snakemake](https://snakemake.readthedocs.io/en/stable/) and I'm trying to understand the difference between the `--cluster` and `--drmaa` flags, both of which are used to submit jobs to compute clusters/nodes.

The docs give a few hints about the advantages of using `--drmaa` [here](http://snakemake.readthedocs.io/en/stable/executable.html):

> If your cluster system supports DRMAA, Snakemake can make use of that
> to increase the control over jobs. E.g. jobs can be cancelled upon
> pressing Ctrl+C, which is not possible with the generic --cluster
> support.

And [here](http://snakemake.readthedocs.io/en/stable/tutorial/additional_features.html):

> If available, DRMAA is preferable over the generic cluster modes
> because it provides better control and error handling.

So I have a conceptual understanding of the advantages of using `--drmaa`. However, I don't consider the above a very complete explanation, and I don't know how these flags are implemented in snakemake under the hood, can someone elaborate?

Note, that although this could be considered a more general programming question, snakemake is predominantly used in bioinformatics; I was persuaded that this question would be considered on-topic because of [this meta-post](https://bioinformatics.meta.stackexchange.com/questions/61/serving-as-a-support-forum-for-specific-tools) and [this answer](https://bioinformatics.meta.stackexchange.com/a/66/104).

**Tags:** snakemake
> **Comments:**
> **username_1**: Hi Chris! What's the actual question here? Seems that DRMAA is an open API describing submission and control of cluster/grid jobs. So if you have a cluster or submission system that has support for it (such as SLURM), `snakemake` will afford you the additional controls as supported by that API -- including as per your question, the ability to pass on a Ctrl+C at your console to the grid -- a scenario that would require use of the `qdel` command on Sun Grid Engine, for example. So if you've infrastructure with a scheduler that has DRMAA support, I can see why you might as well use `--drmaa`.  
> **username_2**: Maybe wait until we've actually poached the snakemake developers :)  
> **username_1**: The code is open source and fairly easy to follow, I found it quite easy to work out how DRMAA jobs were submitted by nosing around the source for 10 minutes. I'm not entirely sure what the question is looking for, though.  

---

## 2 answers

### Answer by username_2 on 2017-06-06. Score: 5 | Accepted answer

I'd always kind of wondered how this worked too, so I took this as an excuse to look into the snakemake code. At the end of the day this becomes a question of (1) how are jobs actually submitted and (2) how is it determined if jobs are done (and then whether they failed)?

For DRMAA, python has a module (appropriately named "drmaa") that wraps around the libdrmaa library that comes with most schedulers. This is a very popular route, for example the Galaxy project uses this for dealing with most clusters (e.g., I use it to connect our internal Galaxy instance to our slurm cluster). The huge benefit here is that DRMAA does magic to allow you to simply submit commands to your cluster without needing to know whether you should execute `qsub` or `srun` or something else. Further, it then provides methods to simply query whether a job is running or not and what its exit status was.

Using the `--cluster` command requires a LOT more magic on the side of snakemake. At the end of the day it creates a shell script that it then submits using the command you provided. Importantly, it includes some secret files in that script that it can then watch for (ever noticed that `.snakemake` directory where you execute a script? That appears to be among the things it's used for.). These are named `{jobid}.jobfinished` and `{jobid}.jobfailed` and one of them will get touched, depending on the exit status of your command/script. Once one of those is there, then snakemake can move on in its DAG (or not, if there's a failure). This is obviously a LOT more to keep track of and it's then not possible for snakemake to cancel running jobs, which is something one can do easily with DRMAA.

> **Comments:**
> **username_1**: Having written a [package for submitting jobs to Sun Grid Engine](https://github.com/SamStudio8/sunblock) without DRMAA, can definitely confirm the magic must be strong here. Though instead of monitoring whether files have been touched, I chose to fire a POST request at the monitoring server. I can see that DRMAA must expose some nice API to get job status without having to do some odd workarounds.  
> **username_2**: Yup, it's more convenient than what I would have come up with too :)  

---

### Answer by username_1 on 2017-06-06. Score: 4

**DRMAA** (Distributed Resource Management Application API) appears to be an open API that describes a specification for submission and management of work submitted to some grid/cluster. [If your scheduler is DRMAA compliant](https://en.wikipedia.org/wiki/DRMAA#Implementations), my guess would be that using `snakemake`'s `--drmaa` flag will afford you the additional controls exposed by that API.

As mentioned in your question with DRMAA support, a Ctrl+C on one's console would be passed on to the grid to kill or stop submitted jobs. A task that would otherwise require use of the `qdel` command on `Sun Grid Engine`, for example.

I could see why the suggestion would be to enable `--drmaa` where possible as it's likely to expose functionality (such as control of jobs, and availability of logs and errors) that makes the submission and management of jobs a little easier for you and `snakemake`. Perhaps someone with `snakemake` experience can explain a little more, but I hope this helps?

For what it's worth, I use Sun Grid Engine without DRMAA and it's quite a pain to get good information out of `qstat`, and its `qacct` log format is quite possibly the worst file format I have ever encountered. I suspect DRMAA provides a nice (or at least more reasonable) API for `snakemake` (and others) to get at this information more readily.

**Appended:**

Under the hood, the flags are parsed by Python's `argparse` module in the `__init__.py`. [A mutually exclusive group of options](https://bitbucket.org/snakemake/snakemake/src/e7bdc7587c6e41113b972e3eba1b9404e741873c/snakemake/__init__.py?fileviewer=file-view-default#__init__.py-747) forces a selection of `--cluster` or `--drmaa` (or `--cluster-sync`).

`--cluster` or `-c` requires you to specify an argument; the command for which to submit a job on your cluster (the example names `qsub`). The `--drmaa` flag just seems to indicate to `snakemake` that DRMAA is to be used, which doesn't change a lot in terms of how the command you run from your console is processed in the `__init__.py` or [`workflow.py`](https://bitbucket.org/snakemake/snakemake/src/e7bdc7587c6e41113b972e3eba1b9404e741873c/snakemake/workflow.py?at=master&fileviewer=file-view-default).

However, when it is time to interface with your scheduler (as specified in [`scheduler.py`](https://bitbucket.org/snakemake/snakemake/src/e7bdc7587c6e41113b972e3eba1b9404e741873c/snakemake/scheduler.py?at=master&fileviewer=file-view-default)), an `elif` statement checks whether you're in some form of cluster mode and the `else` catches the case where you've raised the `--drmaa` flag instead.

Here's where the magic happens, as now your job is submitted with the `DRMAAExecutor` as specified in [executors.py](https://bitbucket.org/snakemake/snakemake/src/e7bdc7587c6e41113b972e3eba1b9404e741873c/snakemake/executors.py?at=master&fileviewer=file-view-default), instead of the default executor.

Without investigating much more, I can see that the `DRMAAExecutor` features some more class attributes and exposes additional functions when compared to the others.

Hope this is of more use? Again, I've only just looked at the code myself but in lieu of a `snakemake` user chiming in, I thought I'd have a nose.

> **Comments:**
> **username_0**: Thanks, but this tells me what I already know; i have a conceptual understanding but i'd like the details on how the 2 different approaches are implemented in snakemake, i'll try to clarify the question a bit  
> **username_1**: Sorry I pre-empted what you might have been asking for as I didn't have much more room to write in the comments. I'll try and expand to address your edited question if I can help!  
> **username_1**: @Chris_Rands I've expanded, is this relevant?  
> **username_0**: Thanks, upvoted, sorry if I wasn't very clear with my question, I just thought this might be useful question for others,i was planning on answering myself if i didn't get any answers, but i've not delved into the source yet  
> **username_1**: @Chris_Rands No problem (and thanks)! I enjoy digging around code bases to see how they work. Between Devon and I, have we found what you were looking for? Happy to chat on this with you if not.  

---

