<p>I understand (see <strong>Slurm array job solution</strong> below), what I can definitely help with is the automated generation of the <code>seqc.yaml</code> file. This avoids the repetition (and risk of error) in copy and pasting. The following one-liner exactly reproduces the <code>seqc.yaml</code> file and needs the <code>chunk.yaml</code> below.</p>
<pre><code>for ((x=10;x&lt;=13;x++)); do while IFS='' read -r a; do echo &quot;<span class="math-container">${a//SM-745-10/SM-745-$</span>{x}}&quot;; done &lt; chunk.yaml &gt;&gt; seqc.yaml; done
</code></pre>
<p>This could be generalised to 3 command lines in bash,</p>
<pre><code>y='SM-745-10'
z='M-745'
for ((x=10;x&lt;=13;x++)); \
do while IFS='' read -r a; do echo &quot;<span class="math-container">${a//${y}/${z}$</span>{x}}&quot;;\
done &lt; chunk.yaml &gt;&gt; seqc.yaml; done
</code></pre>
<p>Here everything from <code>for</code> is a bash one-liner. Obviously <code>((x=10,x&lt;=13;x++))</code> would be <code>((x=1,x&lt;=20;x++))</code> for 20 fastq forward and reverse files. This will ensure the separate files are correctly named in the yaml file, in this example, viz.</p>
<pre><code># first pair
SM-745-10_R1_001.fastq.gz
SM-745-10_R2_001.fastq.gz
# second pair
SM-745-11_R1_001.fastq.gz
SM-745-11_R2_001.fastq.gz
# third pair
SM-745-12_R1_001.fastq.gz
SM-745-12_R2_001.fastq.gz
</code></pre>
<p>The <code>chunk.yaml</code> is,</p>
<pre><code>details:
- algorithm:
    aligner: star
    disambiguate: mm10
    expression_caller:
    - salmon
    - kallisto
    quality_format: standard
    strandedness: auto
    transcriptome_fasta: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/GRCh38.primary_assembly.genome_GPC3CART.fa
    transcriptome_gtf: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/gencode.v29.annotation_GPC3CART.gtf
  analysis: RNA-seq
  description: 10_LI6612_Baseline_8743
  files:
  - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-10_R1_001.fastq.gz
  - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-10_R2_001.fastq.gz
  genome_build: hg38
  metadata:
    panel: Baseline
-
</code></pre>
<hr />
<h3>bcbio</h3>
<p>To summarise the <a href="https://github.com/bcbio/bcbio-nextgen/issues/3719" rel="nofollow noreferrer">response from the Git support team</a> is there's conflict occurring  ...</p>
<ol>
<li>The issue is here for the slurm script ...</li>
</ol>
<pre><code>#SBATCH -p core -n 32
</code></pre>
<p>I assume this is <code>-N 1</code> ,i.e. &gt;= 32 core node, if it was 16 cores per node then <code>-N 2</code>.</p>
<p>However, <strong>bcbio</strong> is taking care of this and the restriction <code>32</code> is conflicting with <code>72</code> from <code>bcbio</code>, i.e. it creates its own Slurm controller script.</p>
<ol start="2">
<li>in <strong>bcbio</strong> yaml</li>
</ol>
<pre><code>    default:
    memory: 30G
    cores: 72
</code></pre>
<p>2a. Firstly <code>72</code> is specified in Slurm's CLI here ... <code>-n 72</code></p>
<p>2b. This isn't needed because,</p>
<pre><code>#SBATCH --mem-per-cpu=40G 
</code></pre>
<p><code>--mem-per-cpu</code>  This is a very recent feature BTW.</p>
<hr />
<h3>Slurm array job solution</h3>
<p>What I considered was using the <code>chunk.yaml</code> above with an array job - this solution isn't needed because <strong>bcbio</strong> is dynamically reading and parallelising <code>seqc.yaml</code> (maybe deploying a wrapper for <code>sinfo</code>, <code>squeue</code>). In the solution below what I'm doing is placing each fastq pair directly into the Slurm queue. This would then parallelise everything directly. Again whilst it would work, it's unnecessary.</p>
<p>Thus for all 20 fastq files assuming <code>SM-745-1_R1_001.fastq.gz</code> ... <code>SM-745-20_R1_001.fastq.gz</code>.</p>
<pre><code>#!/bin/bash

#SBATCH --job-name=bcbiopipeline_GPC3_PDx
#SBATCH -t 1-00:00:00 # 1- is one day. 00:00:00 is a wall clock. Does &quot;1-00:00&quot; work?
#SBATCH -c 1
#SBATCH --array=1-20 # this is now in $SLURM_ARRAY_TASK_ID
#SBATCH --mem-per-cpu=40G
#SBATCH --error=job.%J.err
#SBATCH --output=job.%J.out
#SBATCH --export=ALL

echo &quot;Load bcbio...&quot;
module use /projects/ngs_prodifx/local/software/modules/  
module load bcbio-nextgen/1.2.9 

# SM-745-10 and SM-745 could be assigned as variables y and z as above

while IFS='' read -r a; \
do echo &quot;<span class="math-container">${a//SM-745-10/SM-745-$</span>SLURM_ARRAY_TASK_ID&quot;; \
done &lt; chunk.yaml &gt; seqc.yaml; done
bcbio_nextgen.py seqc.yaml -n 72 -t ipython -s slurm -q core -r t=0-72:00 -r conmem=20 --timeout 3000 --tag &quot;rna&quot;

</code></pre>
<p><strong>Note</strong> the <code>-n 72</code> parameter, i.e. number of cores deployed, will likely be reduced under this solution.</p>
<p><strong>Alternative</strong> The alternative is to issue a slight modification of the bash one-liner described above to produce 3 (or 20) separate yaml files via bash <code>while</code> referring the different fastq paired files and use array job to cycle through them and delete each in turn when done.</p>

CC BY-SA 4.0
2023-10-27T16:05:14.423Z
2023-10-30T17:43:36.270Z
2023-10-30T17:43:36.270Z
M__
<p>Sorry for the cross post but I'm not getting a response on bcbio github:</p>
<p>I'm trying to run bcbio on 20 samples and it's only running one at a time as it steps through the yaml seemingly. I'm using slurm. I was wondering if someone can look at this bcbio snippet and if they see anything wrong?</p>
<p>Slurm script</p>
<pre><code>#!/bin/bash

#SBATCH --job-name=bcbiopipeline_GPC3_PDx
#SBATCH -t 1-00:00
#SBATCH -c 1
#SBATCH -p core -n 32
#SBATCH --mem-per-cpu=40G
#SBATCH --error=job.%J.err
#SBATCH --output=job.%J.out
#SBATCH --export=ALL

echo &quot;Load bcbio...&quot;
module use /projects/ngs_prodifx/local/software/modules/
module load bcbio-nextgen/1.2.9

bcbio_nextgen.py seqc.yaml -n 72 -t ipython -s slurm -q core -r t=0-72:00 -r conmem=20 --timeout 3000 --tag &quot;rna&quot;

</code></pre>
<p>yaml snippet:</p>
<pre><code>resources:
  default:
    memory: 30G
    cores: 72
details:
- algorithm:
    aligner: star
    disambiguate: mm10
    expression_caller:
    - salmon
    - kallisto
    quality_format: standard
    strandedness: auto
    transcriptome_fasta: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/GRCh38.primary_assembly.genome_GPC3CART.fa
    transcriptome_gtf: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/gencode.v29.annotation_GPC3CART.gtf
  analysis: RNA-seq
  description: 10_LI6612_Baseline_8743
  files:
  - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-10_R1_001.fastq.gz
  - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-10_R2_001.fastq.gz
  genome_build: hg38
  metadata:
    panel: Baseline
- algorithm:
    aligner: star
    disambiguate: mm10
    expression_caller:
    - salmon
    - kallisto
    quality_format: standard
    strandedness: auto
    transcriptome_fasta: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/GRCh38.primary_assembly.genome_GPC3CART.fa
    transcriptome_gtf: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/gencode.v29.annotation_GPC3CART.gtf
  analysis: RNA-seq
  description: 11_LI6612_Baseline_8765
  files:
  - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-11_R1_001.fastq.gz
  - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-11_R2_001.fastq.gz
  genome_build: hg38
  metadata:
    panel: Baseline
- algorithm:
    aligner: star
    disambiguate: mm10
    expression_caller:
    - salmon
    - kallisto
    quality_format: standard
    strandedness: auto
    transcriptome_fasta: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/GRCh38.primary_assembly.genome_GPC3CART.fa
    transcriptome_gtf: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/gencode.v29.annotation_GPC3CART.gtf
  analysis: RNA-seq
  description: 12_LI6612_Baseline_8770
  files:
  - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-12_R1_001.fastq.gz
  - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-12_R2_001.fastq.gz
  genome_build: hg38
  metadata:
    panel: Baseline
fc_name: seqc
upload:
  dir: ../output/final
</code></pre>

CC BY-SA 4.0
2023-10-26T15:22:54.083Z
I recommend this is about Slurm rather than bcbio. I would personally do this as a Slurm job array (just my opinion). This is better to resolve via [chat](https://chat.stackexchange.com/rooms/58858/bioinformatics) in my personal opinion. The CLI in the slurm script is key. I can see the Lmod via **use** and **load** probably via EasyBuild.  What I need to do is read about **bcbio** and see how thats put together.
M__
CC BY-SA 4.0
2023-10-26T15:41:43.743Z
You can see the bcbio docs for slurm here: https://bcbio-nextgen.readthedocs.io/en/latest/contents/parallel.html. The setup seems right according to the doc which is why I'm thinking there's an issue with the yaml
t_creasman
CC BY-SA 4.0
2023-10-26T16:39:52.247Z
Okay, I've got it. One solution is an array job.  This will make it directly run in parallel rather than sequentially. *Background* this is EasyBuild on OpenStack (Swift) ...The GitHub request is here: https://github.com/bcbio/bcbio-nextgen/issues/3719
M__
CC BY-SA 4.0
2023-10-26T20:03:15.973Z
@M__ - I reached out to a bcbio expert and it was just a bad sbatch script. You can see his response here: https://github.com/bcbio/bcbio-nextgen/issues/3719
t_creasman
<python><slurm><yaml>
bcbio run not running samples in parallel
# bcbio run not running samples in parallel

(Asked by: username_0 on 2023-10-26)

Sorry for the cross post but I'm not getting a response on bcbio github:
I'm trying to run bcbio on 20 samples and it's only running one at a time as it steps through the yaml seemingly. I'm using slurm. I was wondering if someone can look at this bcbio snippet and if they see anything wrong?
Slurm script
#!/bin/bash

#SBATCH --job-name=bcbiopipeline_GPC3_PDx
#SBATCH -t 1-00:00
#SBATCH -c 1
#SBATCH -p core -n 32
#SBATCH --mem-per-cpu=40G
#SBATCH --error=job.%J.err
#SBATCH --output=job.%J.out
#SBATCH --export=ALL

echo "Load bcbio..."
module use /projects/ngs_prodifx/local/software/modules/
module load bcbio-nextgen/1.2.9

bcbio_nextgen.py seqc.yaml -n 72 -t ipython -s slurm -q core -r t=0-72:00 -r conmem=20 --timeout 3000 --tag "rna"


yaml snippet:
resources:
  default:
    memory: 30G
    cores: 72
details:
- algorithm:
    aligner: star
    disambiguate: mm10
    expression_caller:
    - salmon
    - kallisto
    quality_format: standard
    strandedness: auto
    transcriptome_fasta: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/GRCh38.primary_assembly.genome_GPC3CART.fa
    transcriptome_gtf: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/gencode.v29.annotation_GPC3CART.gtf
  analysis: RNA-seq
  description: 10_LI6612_Baseline_8743
  files:
  - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-10_R1_001.fastq.gz
  - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-10_R2_001.fastq.gz
  genome_build: hg38
  metadata:
    panel: Baseline
- algorithm:
    aligner: star
    disambiguate: mm10
    expression_caller:
    - salmon
    - kallisto
    quality_format: standard
    strandedness: auto
    transcriptome_fasta: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/GRCh38.primary_assembly.genome_GPC3CART.fa
    transcriptome_gtf: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/gencode.v29.annotation_GPC3CART.gtf
  analysis: RNA-seq
  description: 11_LI6612_Baseline_8765
  files:
  - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-11_R1_001.fastq.gz
  - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-11_R2_001.fastq.gz
  genome_build: hg38
  metadata:
    panel: Baseline
- algorithm:
    aligner: star
    disambiguate: mm10
    expression_caller:
    - salmon
    - kallisto
    quality_format: standard
    strandedness: auto
    transcriptome_fasta: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/GRCh38.primary_assembly.genome_GPC3CART.fa
    transcriptome_gtf: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/gencode.v29.annotation_GPC3CART.gtf
  analysis: RNA-seq
  description: 12_LI6612_Baseline_8770
  files:
  - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-12_R1_001.fastq.gz
  - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-12_R2_001.fastq.gz
  genome_build: hg38
  metadata:
    panel: Baseline
fc_name: seqc
upload:
  dir: ../output/final

**Tags:** python, slurm, yaml
> **Comments:**
> **username_1**: I recommend this is about Slurm rather than bcbio. I would personally do this as a Slurm job array (just my opinion). This is better to resolve via [chat](https://chat.stackexchange.com/rooms/58858/bioinformatics) in my personal opinion. The CLI in the slurm script is key. I can see the Lmod via **use** and **load** probably via EasyBuild.  What I need to do is read about **bcbio** and see how thats put together.  
> **username_0**: You can see the bcbio docs for slurm here: https://bcbio-nextgen.readthedocs.io/en/latest/contents/parallel.html. The setup seems right according to the doc which is why I'm thinking there's an issue with the yaml  
> **username_1**: Okay, I've got it. One solution is an array job.  This will make it directly run in parallel rather than sequentially. *Background* this is EasyBuild on OpenStack (Swift) ...The GitHub request is here: https://github.com/bcbio/bcbio-nextgen/issues/3719  
> **username_0**: @M__ - I reached out to a bcbio expert and it was just a bad sbatch script. You can see his response here: https://github.com/bcbio/bcbio-nextgen/issues/3719  

---

## 1 answers

### Answer by username_1 on 2023-10-27. Score: 0

I understand (see Slurm array job solution below), what I can definitely help with is the automated generation of the seqc.yaml file. This avoids the repetition (and risk of error) in copy and pasting. The following one-liner exactly reproduces the seqc.yaml file and needs the chunk.yaml below.
for ((x=10;x<=13;x++)); do while IFS='' read -r a; do echo "${a//SM-745-10/SM-745-${x}}"; done < chunk.yaml >> seqc.yaml; done

This could be generalised to 3 command lines in bash,
y='SM-745-10'
z='M-745'
for ((x=10;x<=13;x++)); \
do while IFS='' read -r a; do echo "${a//${y}/${z}${x}}";\
done < chunk.yaml >> seqc.yaml; done

Here everything from for is a bash one-liner. Obviously ((x=10,x<=13;x++)) would be ((x=1,x<=20;x++)) for 20 fastq forward and reverse files. This will ensure the separate files are correctly named in the yaml file, in this example, viz.
# first pair
SM-745-10_R1_001.fastq.gz
SM-745-10_R2_001.fastq.gz
# second pair
SM-745-11_R1_001.fastq.gz
SM-745-11_R2_001.fastq.gz
# third pair
SM-745-12_R1_001.fastq.gz
SM-745-12_R2_001.fastq.gz

The chunk.yaml is,
details:
- algorithm:
    aligner: star
    disambiguate: mm10
    expression_caller:
    - salmon
    - kallisto
    quality_format: standard
    strandedness: auto
    transcriptome_fasta: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/GRCh38.primary_assembly.genome_GPC3CART.fa
    transcriptome_gtf: /swiftcache/ngs/oncology/analysis/todd_creasy/working/reference/gencode.v29.annotation_GPC3CART.gtf
  analysis: RNA-seq
  description: 10_LI6612_Baseline_8743
  files:
  - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-10_R1_001.fastq.gz
  - /swiftcache/ngs/oncology/analysis/todd_creasy/working/fastq/SM-745/SM-745-10_R2_001.fastq.gz
  genome_build: hg38
  metadata:
    panel: Baseline
-


bcbio
To summarise the response from the Git support team is there's conflict occurring  ...

The issue is here for the slurm script ...

#SBATCH -p core -n 32

I assume this is -N 1 ,i.e. >= 32 core node, if it was 16 cores per node then -N 2.
However, bcbio is taking care of this and the restriction 32 is conflicting with 72 from bcbio, i.e. it creates its own Slurm controller script.

in bcbio yaml

    default:
    memory: 30G
    cores: 72

2a. Firstly 72 is specified in Slurm's CLI here ... -n 72
2b. This isn't needed because,
#SBATCH --mem-per-cpu=40G 

--mem-per-cpu  This is a very recent feature BTW.

Slurm array job solution
What I considered was using the chunk.yaml above with an array job - this solution isn't needed because bcbio is dynamically reading and parallelising seqc.yaml (maybe deploying a wrapper for sinfo, squeue). In the solution below what I'm doing is placing each fastq pair directly into the Slurm queue. This would then parallelise everything directly. Again whilst it would work, it's unnecessary.
Thus for all 20 fastq files assuming SM-745-1_R1_001.fastq.gz ... SM-745-20_R1_001.fastq.gz.
#!/bin/bash

#SBATCH --job-name=bcbiopipeline_GPC3_PDx
#SBATCH -t 1-00:00:00 # 1- is one day. 00:00:00 is a wall clock. Does "1-00:00" work?
#SBATCH -c 1
#SBATCH --array=1-20 # this is now in $SLURM_ARRAY_TASK_ID
#SBATCH --mem-per-cpu=40G
#SBATCH --error=job.%J.err
#SBATCH --output=job.%J.out
#SBATCH --export=ALL

echo "Load bcbio..."
module use /projects/ngs_prodifx/local/software/modules/  
module load bcbio-nextgen/1.2.9 

# SM-745-10 and SM-745 could be assigned as variables y and z as above

while IFS='' read -r a; \
do echo "${a//SM-745-10/SM-745-$SLURM_ARRAY_TASK_ID"; \
done < chunk.yaml > seqc.yaml; done
bcbio_nextgen.py seqc.yaml -n 72 -t ipython -s slurm -q core -r t=0-72:00 -r conmem=20 --timeout 3000 --tag "rna"


Note the -n 72 parameter, i.e. number of cores deployed, will likely be reduced under this solution.
Alternative The alternative is to issue a slight modification of the bash one-liner described above to produce 3 (or 20) separate yaml files via bash while referring the different fastq paired files and use array job to cycle through them and delete each in turn when done.


---

