<p>Slurm is killing the process because it's using more memory per core than the default allocation (that's what <code>slurmstepd: error: Exceeded step memory limit at some point</code> means). I've never used pyPiper (like seemingly most people, I use snakemake), but presuming that you have it running either <code>srun</code> or <code>sbatch</code> at some point then specify something like <code>--mem-per-cpu 10G</code>.</p>

CC BY-SA 3.0
2017-12-15T15:12:39.587Z
Actually I tried that and it still does not work. Will try to write to the cluster support
Nikita Vlasenko
CC BY-SA 3.0
2017-12-15T15:22:35.497Z
You might need a larger value than 10 gigs. You can use `scontrol` to see the exact threshold getting crossed that's causing slurm to kill the job.
Devon Ryan
CC BY-SA 3.0
2017-12-15T16:53:31.197Z
Awesome. We increased it to 20Gb and it run successfully.
Nikita Vlasenko
CC BY-SA 3.0
2017-12-14T23:17:52.967Z
2017-12-14T23:17:52.967Z
Devon Ryan
<p>I am running a script file on a cluster with <code>slurm</code> through <code>sbatch</code>. Locally it runs perfectly well on 2 different linux versions (<code>linux mint</code>, <code>ubuntu 12.04</code>), however, it returns <code>-9</code> error code while running on a cluster which is <code>CentOS 7.3</code>. When we are using <code>Pypiper</code> and run the whole <code>.py</code> script file, it fails when it starts running <code>trimmomatic</code> program</p>

<p><a href="http://www.usadellab.org/cms/?page=trimmomatic" rel="nofollow noreferrer">http://www.usadellab.org/cms/?page=trimmomatic</a></p>

<p>However, when running just <code>trimmomatic</code> program step by itself it does not fail and finishes successfully. We have a suspicion that something may be wrong with how <code>Pypiper</code> runs on the cluster, however, feeling rather stuck.
Any suggestions would be greatly appreciated. </p>

<p>Here is the error that is printed out:</p>

<pre><code>ILLUMINACLIP: Using 2 prefix pairs, 22 forward/reverse sequences, 0 
forward only sequences, 0 reverse only sequences
&lt;/pre&gt;
Process 109391 returned: (-9). Elapsed: 0:00:51. Peak memory: (Process: 
5.211GB; Pipeline: 0.0GB)

### Pipeline failed at:  (12-14 15:50:47) elapsed:0:23:09 _TIME_

Total time: 0:23:09

Change status from running to failed
Traceback (most recent call last):
  File "...../salmon_rna_seq.py", line 344, in &lt;module&gt;
sys.exit(main())
File "...../salmon_rna_seq.py", line 304, in main
pipeline.process()      
File "...../salmon_rna_seq.py", line 277, in process
self.trim_reads()      
File "...../salmon_rna_seq.py", line 195, in trim_reads
self.pipe_manager.run(cmd, self.sample_files['r1']['fastq']['trimmed_paired'])
File "...../pypiper.py", line 508, in run
process_return_code, local_maxmem = self.callprint(cmd, shell, nofail, container)  # Run command
File "...../pypiper.py", line 691, in callprint
self.fail_pipeline(e)
File "...../pypiper.py", line 1111, in fail_pipeline
raise e
OSError: Subprocess returned nonzero result.

Pypiper terminating spawned child process 108475...
child process terminated
slurmstepd: error: Exceeded step memory limit at some point.
</code></pre>

<p>The last line may be informative. Although, we are still not sure what we could possibly do.</p>

<rna-seq>
Pipeline (Pypiper) returns error, but individual command works fine
# Pipeline (Pypiper) returns error, but individual command works fine

(Asked by: username_0 on 2017-12-14)

I am running a script file on a cluster with slurm through sbatch. Locally it runs perfectly well on 2 different linux versions (linux mint, ubuntu 12.04), however, it returns -9 error code while running on a cluster which is CentOS 7.3. When we are using Pypiper and run the whole .py script file, it fails when it starts running trimmomatic program
http://www.usadellab.org/cms/?page=trimmomatic
However, when running just trimmomatic program step by itself it does not fail and finishes successfully. We have a suspicion that something may be wrong with how Pypiper runs on the cluster, however, feeling rather stuck.
Any suggestions would be greatly appreciated. 
Here is the error that is printed out:
ILLUMINACLIP: Using 2 prefix pairs, 22 forward/reverse sequences, 0 
forward only sequences, 0 reverse only sequences

Process 109391 returned: (-9). Elapsed: 0:00:51. Peak memory: (Process: 
5.211GB; Pipeline: 0.0GB)

### Pipeline failed at:  (12-14 15:50:47) elapsed:0:23:09 _TIME_

Total time: 0:23:09

Change status from running to failed
Traceback (most recent call last):
  File "...../salmon_rna_seq.py", line 344, in 
sys.exit(main())
File "...../salmon_rna_seq.py", line 304, in main
pipeline.process()      
File "...../salmon_rna_seq.py", line 277, in process
self.trim_reads()      
File "...../salmon_rna_seq.py", line 195, in trim_reads
self.pipe_manager.run(cmd, self.sample_files['r1']['fastq']['trimmed_paired'])
File "...../pypiper.py", line 508, in run
process_return_code, local_maxmem = self.callprint(cmd, shell, nofail, container)  # Run command
File "...../pypiper.py", line 691, in callprint
self.fail_pipeline(e)
File "...../pypiper.py", line 1111, in fail_pipeline
raise e
OSError: Subprocess returned nonzero result.

Pypiper terminating spawned child process 108475...
child process terminated
slurmstepd: error: Exceeded step memory limit at some point.

The last line may be informative. Although, we are still not sure what we could possibly do.

**Tags:** rna-seq

---

## 1 answers

### Answer by username_1 on 2017-12-14. Score: 3 | Accepted answer

Slurm is killing the process because it's using more memory per core than the default allocation (that's what `slurmstepd: error: Exceeded step memory limit at some point` means). I've never used pyPiper (like seemingly most people, I use snakemake), but presuming that you have it running either `srun` or `sbatch` at some point then specify something like `--mem-per-cpu 10G`.

> **Comments:**
> **username_0**: Actually I tried that and it still does not work. Will try to write to the cluster support  
> **username_1**: You might need a larger value than 10 gigs. You can use `scontrol` to see the exact threshold getting crossed that's causing slurm to kill the job.  
> **username_0**: Awesome. We increased it to 20Gb and it run successfully.  

---

