<p>I have some time now, so I'm going to take a shot at this, @M__</p>
<p>You need to run spades per read pair. When you specify <code>*_1.fastq.gz</code> as the <code>-1</code> param, you're giving it <em>every</em> <code>_1</code> file at once, not one <code>_1</code> file per spades command.</p>
<p>I'd recommend this:</p>
<pre><code>for smpl in <span class="math-container">$(ls *_1.fastq.gz | xargs -I infile basename infile _1.fastq.gz)
do
    cat &gt; "./$</span>{smpl}_spades.slurm&quot; &lt;&lt;EOF
#!/bin/bash  
 
#SBATCH --job-name=pangenome_spades.${smpl}  
#SBATCH --account=lilianasalvador
#SBATCH --partition=standard
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --mem-per-cpu=5gb
#SBATCH --time=02:00:00

spades.py -1 <span class="math-container">${smpl}_1.fastq.gz -2 $</span>{smpl}_2.fastq.gz -o groups/lilianasalvador/noah_100/SPAdes_output_${smpl} -t 16 --careful
EOF
done
</code></pre>
<p>Run a sanity check on a few of the <code>.slurm</code> files, then</p>
<pre><code>ls *.slurm | xargs -I job_file sbatch job_file
</code></pre>

CC BY-SA 4.0
2024-05-30T03:16:08.877Z
Cool, thanks. I was going to solve this via a formal Slurm array job, but this will work.
M__
CC BY-SA 4.0
2024-05-30T13:26:01.230Z
Please do add an array-based answer, I'm not familiar with arrays but would love to learn
Ram RS
CC BY-SA 4.0
2024-06-06T19:00:04.813Z
Thank you! This worked but we're only getting one contigs.fasta file when we run. Should we be getting a contigs.fasta per sequence pair or one for all the sequences we ran?
Lucy Kelly
CC BY-SA 4.0
2024-06-07T20:46:22.140Z
I am not familiar with spades but going by M__'s answer, you need something at the `-o` param as well. I'll edit my answer but you should test running 2 samples at the same time before running all of the scripts concurrently.
Ram RS
CC BY-SA 4.0
2024-05-30T01:05:52.363Z
2024-06-07T20:47:31.277Z
2024-06-07T20:47:31.277Z
Ram RS
<p>The <em>job array</em> is a formal approach to solving this via SLURM</p>
<p>What I am not clear about is the number of files <code>SRR8599992_1.fastq.gz</code> <code>SRR8599992_2.fastq.gaz</code> I originally thought the final number of files was around 100, I've guessed at 120</p>
<pre><code>#SBATCH --job-name=pangenome_spade
#SBATCH --account=lilianasalvador
#SBATCH --ntasks=120
#SBATCH --cpus-per-task=16 
#SBATCH --mem-per-cpu=5gb  
#SBATCH --time=02:00:00 
#SBATCH --array=599992-5600100

module load python/3.9  
module load spades/3.15.0-hpc 

spades.py -1 'SRR'<span class="math-container">${SLURM_ARRAY_TASK_ID}'_1.fastq.gz' -2 'SRR'$</span>{<span class="math-container">$SLURM_ARRAY_TASK_ID}'_2.fastq.gz' -o /groups/lilianasalvador/noah_100/'SPAdes_output_'$</span>{SLURM_ARRAY_TASK_ID}'.fastq' -t 16 --careful 
</code></pre>
<p>I would test this via an <code>echo</code> test script writing to a specific file to make sure the <code>spades.py</code> command produces the correct command line for each iteration.</p>
<p>Things to note ...</p>
<ul>
<li>Array jobs is specified by the <code>$SLURM_ARRAY_TASK_ID</code> variable and <code>#SBATCH --array=599992-5600100</code></li>
<li><code>ntasks</code> is the total number of spades assemblies you are performing here I guessed at 120</li>
<li>You are parallelising via <code>-t 16</code> therefore this must be reflected in your submission job via <code>--cpus-per-task</code></li>
<li>The <em>ouput</em> file <em>MUST</em> use <code>$SLURM_ARRAY_TASK_ID</code> otherwise you will continually override your out file.</li>
</ul>
<hr />
<p>This is not an optimal SLURM job because ...</p>
<ul>
<li><code>-t 16</code> you need to check with your HPC engineer, this might take a long time to get through the queue and an alternative architecture to permit 16 CPUs can get onto the cluster. HTC users will jump ahead of you, so on a hypervisor setup you can specify a separate cluster for your work. This is not well explained but hope it makes sense.</li>
<li>Shifting to high throughput computing (HTC) could be considered here to get this through a lot quicker, e.g. this is 120 sequential assemblies, 10 sequential assemblies for 12 separate submissions would be a lot quicker (you just alter the &quot;array job&quot; inputs range.</li>
</ul>

CC BY-SA 4.0
2024-06-01T17:10:17.953Z
2024-06-02T03:22:50.020Z
2024-06-02T03:22:50.020Z
M__
<p>I am trying to run a batch script to assemble sequences using the spades.py command on HPC. I tested the script with one sequence (1 forward and 1 reverse read) and this worked. I have 100 sequences in total (200 files) and am trying to assemble all of them without having to resubmit the script each time. This is the original script that worked:</p>
<pre><code>#!/bin/bash  
 
#SBATCH --job-name=pangenome_spades  
#SBATCH --account=lilianasalvador  
#SBATCH --partition=standard  
#SBATCH --nodes=1  
#SBATCH --ntasks=1  
#SBATCH --mem-per-cpu=5gb  
#SBATCH --time=01:00:00  
 
spades.py -1 SRR8599992_1.fastq.gz -2 SRR8599992_2.fastq.gz -o /home/u32/lucykelly/spades_test -t 16 --careful 
</code></pre>
<p>This is the script that I've been trying to run and the error it's giving me:</p>
<pre><code>#!/bin/bash  
 
#SBATCH --job-name=pangenome_spades  
#SBATCH --account=lilianasalvador  
#SBATCH --partition=standard  
#SBATCH --nodes=1  
#SBATCH --ntasks=1  
#SBATCH --mem-per-cpu=5gb  
#SBATCH --time=02:00:00  
 
module load python/3.9  
module load spades/3.15.0-hpc 

spades.py -1 *_1.fastq.gz -2 *_2.fastq.gz -o /groups/lilianasalvador/noah_100/SPAdes_output -t 16 --careful 

SPAdes genome assembler v3.15.0-hpc 
 
Usage: spades.py [options] -o &lt;output_dir&gt; 
spades.py: error: Please specify option (e.g. -1, -2, -s, etc)) for the following paths: SRR8599995_1.fastq.gz, SRR8599997_1.fastq.gz, SRR8599998_1.fastq.gz, SRR8600006_1.fastq.gz, SRR8600007_1.fastq.gz, SRR8600008_1.fastq.gz, SRR8600014_1.fastq.gz, SRR8600015_1.fastq.gz, SRR8600020_1.fastq.gz, SRR8600022_1.fastq.gz, SRR8600026_1.fastq.gz, SRR8600031_1.fastq.gz, SRR8600044_1.fastq.gz, SRR8600045_1.fastq.gz, SRR8600048_1.fastq.gz, SRR8600049_1.fastq.gz, SRR8600052_1.fastq.gz, SRR8600055_1.fastq.gz, SRR8600058_1.fastq.gz, SRR8600059_1.fastq.gz, SRR8600064_1.fastq.gz, SRR8600070_1.fastq.gz, SRR8600072_1.fastq.gz, SRR8600075_1.fastq.gz, SRR8600079_1.fastq.gz, SRR8600092_1.fastq.gz, SRR8600093_1.fastq.gz, SRR8600098_1.fastq.gz, SRR8600100_1.fastq.gz, SRR8600101_1.fastq.gz, SRR8600102_1.fastq.gz, SRR8600103_1.fastq.gz, SRR8600105_1.fastq.gz, SRR8600110_1.fastq.gz, SRR8600113_1.fastq.gz, SRR8600117_1.fastq.gz, SRR8600121_1.fastq.gz, SRR8600122_1.fastq.gz, SRR8600123_1.fastq.gz, SRR8600128_1.fastq.gz, SRR8600133_1.fastq.gz, SRR8600134_1.fastq.gz, SRR8600136_1.fastq.gz, SRR8600145_1.fastq.gz, SRR8600147_1.fastq.gz, SRR8600160_1.fastq.gz, SRR8600164_1.fastq.gz, SRR8600166_1.fastq.gz, SRR8600167_1.fastq.gz, SRR8600174_1.fastq.gz, SRR8600182_1.fastq.gz, SRR8600184_1.fastq.gz, SRR8600185_1.fastq.gz, SRR8600188_1.fastq.gz, SRR8600189_1.fastq.gz, SRR8600190_1.fastq.gz, SRR8600193_1.fastq.gz, SRR8600194_1.fastq.gz, SRR8600216_1.fastq.gz, SRR8600222_1.fastq.gz, SRR8600223_1.fastq.gz, SRR8600226_1.fastq.gz, SRR8600228_1.fastq.gz, SRR8600235_1.fastq.gz, SRR8600236_1.fastq.gz, SRR8600243_1.fastq.gz, SRR8600244_1.fastq.gz, SRR8600251_1.fastq.gz, SRR8600252_1.fastq.gz, SRR8600258_1.fastq.gz, SRR8600259_1.fastq.gz, SRR8600264_1.fastq.gz, SRR8600267_1.fastq.gz, SRR8600268_1.fastq.gz, SRR8600270_1.fastq.gz, SRR8600272_1.fastq.gz, SRR8600274_1.fastq.gz, SRR8600277_1.fastq.gz, SRR8600278_1.fastq.gz, SRR8600281_1.fastq.gz, SRR8600282_1.fastq.gz, SRR8600283_1.fastq.gz, SRR8600285_1.fastq.gz, SRR8600287_1.fastq.gz, SRR8600288_1.fastq.gz, SRR8600291_1.fastq.gz, SRR8600294_1.fastq.gz, SRR8600296_1.fastq.gz, SRR8600302_1.fastq.gz, SRR8600308_1.fastq.gz, SRR8600309_1.fastq.gz, SRR8600311_1.fastq.gz, SRR8600319_1.fastq.gz, SRR8600327_1.fastq.gz, SRR8600328_1.fastq.gz, SRR8600329_1.fastq.gz, SRR8600331_1.fastq.gz, SRR8600339_1.fastq.gz, SRR8599995_2.fastq.gz, SRR8599997_2.fastq.gz, SRR8599998_2.fastq.gz, SRR8600006_2.fastq.gz, SRR8600007_2.fastq.gz, SRR8600008_2.fastq.gz, SRR8600014_2.fastq.gz, SRR8600015_2.fastq.gz, SRR8600020_2.fastq.gz, SRR8600022_2.fastq.gz, SRR8600026_2.fastq.gz, SRR8600031_2.fastq.gz, SRR8600044_2.fastq.gz, SRR8600045_2.fastq.gz, SRR8600048_2.fastq.gz, SRR8600049_2.fastq.gz, SRR8600052_2.fastq.gz, SRR8600055_2.fastq.gz, SRR8600058_2.fastq.gz, SRR8600059_2.fastq.gz, SRR8600064_2.fastq.gz, SRR8600070_2.fastq.gz, SRR8600072_2.fastq.gz, SRR8600075_2.fastq.gz, SRR8600079_2.fastq.gz, SRR8600092_2.fastq.gz, SRR8600093_2.fastq.gz, SRR8600098_2.fastq.gz, SRR8600100_2.fastq.gz, SRR8600101_2.fastq.gz, SRR8600102_2.fastq.gz, SRR8600103_2.fastq.gz, SRR8600105_2.fastq.gz, SRR8600110_2.fastq.gz, SRR8600113_2.fastq.gz, SRR8600117_2.fastq.gz, SRR8600121_2.fastq.gz, SRR8600122_2.fastq.gz, SRR8600123_2.fastq.gz, SRR8600128_2.fastq.gz, SRR8600133_2.fastq.gz, SRR8600134_2.fastq.gz, SRR8600136_2.fastq.gz, SRR8600145_2.fastq.gz, SRR8600147_2.fastq.gz, SRR8600160_2.fastq.gz, SRR8600164_2.fastq.gz, SRR8600166_2.fastq.gz, SRR8600167_2.fastq.gz, SRR8600174_2.fastq.gz, SRR8600182_2.fastq.gz, SRR8600184_2.fastq.gz, SRR8600185_2.fastq.gz, SRR8600188_2.fastq.gz, SRR8600189_2.fastq.gz, SRR8600190_2.fastq.gz, SRR8600193_2.fastq.gz, SRR8600194_2.fastq.gz, SRR8600216_2.fastq.gz, SRR8600222_2.fastq.gz, SRR8600223_2.fastq.gz, SRR8600226_2.fastq.gz, SRR8600228_2.fastq.gz, SRR8600235_2.fastq.gz, SRR8600236_2.fastq.gz, SRR8600243_2.fastq.gz, SRR8600244_2.fastq.gz, SRR8600251_2.fastq.gz, SRR8600252_2.fastq.gz, SRR8600258_2.fastq.gz, SRR8600259_2.fastq.gz, SRR8600264_2.fastq.gz, SRR8600267_2.fastq.gz, SRR8600268_2.fastq.gz, SRR8600270_2.fastq.gz, SRR8600272_2.fastq.gz, SRR8600274_2.fastq.gz, SRR8600277_2.fastq.gz, SRR8600278_2.fastq.gz, SRR8600281_2.fastq.gz, SRR8600282_2.fastq.gz, SRR8600283_2.fastq.gz, SRR8600285_2.fastq.gz, SRR8600287_2.fastq.gz, SRR8600288_2.fastq.gz, SRR8600291_2.fastq.gz, SRR8600294_2.fastq.gz, SRR8600296_2.fastq.gz, SRR8600302_2.fastq.gz, SRR8600308_2.fastq.gz, SRR8600309_2.fastq.gz, SRR8600311_2.fastq.gz, SRR8600319_2.fastq.gz, SRR8600327_2.fastq.gz, SRR8600328_2.fastq.gz, SRR8600329_2.fastq.gz, SRR8600331_2.fastq.gz, SRR8600339_2.fastq.gz 
<span class="math-container">```</span>
</code></pre>

CC BY-SA 4.0
2024-05-29T21:07:12.943Z
This is easy. I'll respond over the weekend.
M__
CC BY-SA 4.0
2024-06-08T08:25:46.983Z
There is brand new spades release with a bunch of improvements. You may check if it is compatible with your HPC setup. https://github.com/ablab/spades/releases/tag/v4.0.0
darked89
<python><genome><hpc>
Running multiple sequence using spades.py on an HPC
# Running multiple sequence using spades.py on an HPC

(Asked by: username_0 on 2024-05-29)

I am trying to run a batch script to assemble sequences using the spades.py command on HPC. I tested the script with one sequence (1 forward and 1 reverse read) and this worked. I have 100 sequences in total (200 files) and am trying to assemble all of them without having to resubmit the script each time. This is the original script that worked:
#!/bin/bash  
 
#SBATCH --job-name=pangenome_spades  
#SBATCH --account=lilianasalvador  
#SBATCH --partition=standard  
#SBATCH --nodes=1  
#SBATCH --ntasks=1  
#SBATCH --mem-per-cpu=5gb  
#SBATCH --time=01:00:00  
 
spades.py -1 SRR8599992_1.fastq.gz -2 SRR8599992_2.fastq.gz -o /home/u32/lucykelly/spades_test -t 16 --careful 

This is the script that I've been trying to run and the error it's giving me:
#!/bin/bash  
 
#SBATCH --job-name=pangenome_spades  
#SBATCH --account=lilianasalvador  
#SBATCH --partition=standard  
#SBATCH --nodes=1  
#SBATCH --ntasks=1  
#SBATCH --mem-per-cpu=5gb  
#SBATCH --time=02:00:00  
 
module load python/3.9  
module load spades/3.15.0-hpc 

spades.py -1 *_1.fastq.gz -2 *_2.fastq.gz -o /groups/lilianasalvador/noah_100/SPAdes_output -t 16 --careful 

SPAdes genome assembler v3.15.0-hpc 
 
Usage: spades.py [options] -o  
spades.py: error: Please specify option (e.g. -1, -2, -s, etc)) for the following paths: SRR8599995_1.fastq.gz, SRR8599997_1.fastq.gz, SRR8599998_1.fastq.gz, SRR8600006_1.fastq.gz, SRR8600007_1.fastq.gz, SRR8600008_1.fastq.gz, SRR8600014_1.fastq.gz, SRR8600015_1.fastq.gz, SRR8600020_1.fastq.gz, SRR8600022_1.fastq.gz, SRR8600026_1.fastq.gz, SRR8600031_1.fastq.gz, SRR8600044_1.fastq.gz, SRR8600045_1.fastq.gz, SRR8600048_1.fastq.gz, SRR8600049_1.fastq.gz, SRR8600052_1.fastq.gz, SRR8600055_1.fastq.gz, SRR8600058_1.fastq.gz, SRR8600059_1.fastq.gz, SRR8600064_1.fastq.gz, SRR8600070_1.fastq.gz, SRR8600072_1.fastq.gz, SRR8600075_1.fastq.gz, SRR8600079_1.fastq.gz, SRR8600092_1.fastq.gz, SRR8600093_1.fastq.gz, SRR8600098_1.fastq.gz, SRR8600100_1.fastq.gz, SRR8600101_1.fastq.gz, SRR8600102_1.fastq.gz, SRR8600103_1.fastq.gz, SRR8600105_1.fastq.gz, SRR8600110_1.fastq.gz, SRR8600113_1.fastq.gz, SRR8600117_1.fastq.gz, SRR8600121_1.fastq.gz, SRR8600122_1.fastq.gz, SRR8600123_1.fastq.gz, SRR8600128_1.fastq.gz, SRR8600133_1.fastq.gz, SRR8600134_1.fastq.gz, SRR8600136_1.fastq.gz, SRR8600145_1.fastq.gz, SRR8600147_1.fastq.gz, SRR8600160_1.fastq.gz, SRR8600164_1.fastq.gz, SRR8600166_1.fastq.gz, SRR8600167_1.fastq.gz, SRR8600174_1.fastq.gz, SRR8600182_1.fastq.gz, SRR8600184_1.fastq.gz, SRR8600185_1.fastq.gz, SRR8600188_1.fastq.gz, SRR8600189_1.fastq.gz, SRR8600190_1.fastq.gz, SRR8600193_1.fastq.gz, SRR8600194_1.fastq.gz, SRR8600216_1.fastq.gz, SRR8600222_1.fastq.gz, SRR8600223_1.fastq.gz, SRR8600226_1.fastq.gz, SRR8600228_1.fastq.gz, SRR8600235_1.fastq.gz, SRR8600236_1.fastq.gz, SRR8600243_1.fastq.gz, SRR8600244_1.fastq.gz, SRR8600251_1.fastq.gz, SRR8600252_1.fastq.gz, SRR8600258_1.fastq.gz, SRR8600259_1.fastq.gz, SRR8600264_1.fastq.gz, SRR8600267_1.fastq.gz, SRR8600268_1.fastq.gz, SRR8600270_1.fastq.gz, SRR8600272_1.fastq.gz, SRR8600274_1.fastq.gz, SRR8600277_1.fastq.gz, SRR8600278_1.fastq.gz, SRR8600281_1.fastq.gz, SRR8600282_1.fastq.gz, SRR8600283_1.fastq.gz, SRR8600285_1.fastq.gz, SRR8600287_1.fastq.gz, SRR8600288_1.fastq.gz, SRR8600291_1.fastq.gz, SRR8600294_1.fastq.gz, SRR8600296_1.fastq.gz, SRR8600302_1.fastq.gz, SRR8600308_1.fastq.gz, SRR8600309_1.fastq.gz, SRR8600311_1.fastq.gz, SRR8600319_1.fastq.gz, SRR8600327_1.fastq.gz, SRR8600328_1.fastq.gz, SRR8600329_1.fastq.gz, SRR8600331_1.fastq.gz, SRR8600339_1.fastq.gz, SRR8599995_2.fastq.gz, SRR8599997_2.fastq.gz, SRR8599998_2.fastq.gz, SRR8600006_2.fastq.gz, SRR8600007_2.fastq.gz, SRR8600008_2.fastq.gz, SRR8600014_2.fastq.gz, SRR8600015_2.fastq.gz, SRR8600020_2.fastq.gz, SRR8600022_2.fastq.gz, SRR8600026_2.fastq.gz, SRR8600031_2.fastq.gz, SRR8600044_2.fastq.gz, SRR8600045_2.fastq.gz, SRR8600048_2.fastq.gz, SRR8600049_2.fastq.gz, SRR8600052_2.fastq.gz, SRR8600055_2.fastq.gz, SRR8600058_2.fastq.gz, SRR8600059_2.fastq.gz, SRR8600064_2.fastq.gz, SRR8600070_2.fastq.gz, SRR8600072_2.fastq.gz, SRR8600075_2.fastq.gz, SRR8600079_2.fastq.gz, SRR8600092_2.fastq.gz, SRR8600093_2.fastq.gz, SRR8600098_2.fastq.gz, SRR8600100_2.fastq.gz, SRR8600101_2.fastq.gz, SRR8600102_2.fastq.gz, SRR8600103_2.fastq.gz, SRR8600105_2.fastq.gz, SRR8600110_2.fastq.gz, SRR8600113_2.fastq.gz, SRR8600117_2.fastq.gz, SRR8600121_2.fastq.gz, SRR8600122_2.fastq.gz, SRR8600123_2.fastq.gz, SRR8600128_2.fastq.gz, SRR8600133_2.fastq.gz, SRR8600134_2.fastq.gz, SRR8600136_2.fastq.gz, SRR8600145_2.fastq.gz, SRR8600147_2.fastq.gz, SRR8600160_2.fastq.gz, SRR8600164_2.fastq.gz, SRR8600166_2.fastq.gz, SRR8600167_2.fastq.gz, SRR8600174_2.fastq.gz, SRR8600182_2.fastq.gz, SRR8600184_2.fastq.gz, SRR8600185_2.fastq.gz, SRR8600188_2.fastq.gz, SRR8600189_2.fastq.gz, SRR8600190_2.fastq.gz, SRR8600193_2.fastq.gz, SRR8600194_2.fastq.gz, SRR8600216_2.fastq.gz, SRR8600222_2.fastq.gz, SRR8600223_2.fastq.gz, SRR8600226_2.fastq.gz, SRR8600228_2.fastq.gz, SRR8600235_2.fastq.gz, SRR8600236_2.fastq.gz, SRR8600243_2.fastq.gz, SRR8600244_2.fastq.gz, SRR8600251_2.fastq.gz, SRR8600252_2.fastq.gz, SRR8600258_2.fastq.gz, SRR8600259_2.fastq.gz, SRR8600264_2.fastq.gz, SRR8600267_2.fastq.gz, SRR8600268_2.fastq.gz, SRR8600270_2.fastq.gz, SRR8600272_2.fastq.gz, SRR8600274_2.fastq.gz, SRR8600277_2.fastq.gz, SRR8600278_2.fastq.gz, SRR8600281_2.fastq.gz, SRR8600282_2.fastq.gz, SRR8600283_2.fastq.gz, SRR8600285_2.fastq.gz, SRR8600287_2.fastq.gz, SRR8600288_2.fastq.gz, SRR8600291_2.fastq.gz, SRR8600294_2.fastq.gz, SRR8600296_2.fastq.gz, SRR8600302_2.fastq.gz, SRR8600308_2.fastq.gz, SRR8600309_2.fastq.gz, SRR8600311_2.fastq.gz, SRR8600319_2.fastq.gz, SRR8600327_2.fastq.gz, SRR8600328_2.fastq.gz, SRR8600329_2.fastq.gz, SRR8600331_2.fastq.gz, SRR8600339_2.fastq.gz 
```

**Tags:** python, genome, hpc
> **Comments:**
> **username_1**: This is easy. I'll respond over the weekend.  
> **username_2**: There is brand new spades release with a bunch of improvements. You may check if it is compatible with your HPC setup. https://github.com/ablab/spades/releases/tag/v4.0.0  

---

## 2 answers

### Answer by username_3 on 2024-05-30. Score: 5

I have some time now, so I'm going to take a shot at this, @M__
You need to run spades per read pair. When you specify *_1.fastq.gz as the -1 param, you're giving it every _1 file at once, not one _1 file per spades command.
I'd recommend this:
for smpl in $(ls *_1.fastq.gz | xargs -I infile basename infile _1.fastq.gz)
do
    cat > "./${smpl}_spades.slurm" <${smpl}_1.fastq.gz -2 ${smpl}_2.fastq.gz -o groups/lilianasalvador/noah_100/SPAdes_output_${smpl} -t 16 --careful
EOF
done

Run a sanity check on a few of the .slurm files, then
ls *.slurm | xargs -I job_file sbatch job_file

> **Comments:**
> **username_1**: Cool, thanks. I was going to solve this via a formal Slurm array job, but this will work.  
> **username_3**: Please do add an array-based answer, I'm not familiar with arrays but would love to learn  
> **username_0**: Thank you! This worked but we're only getting one contigs.fasta file when we run. Should we be getting a contigs.fasta per sequence pair or one for all the sequences we ran?  
> **username_3**: I am not familiar with spades but going by M__'s answer, you need something at the `-o` param as well. I'll edit my answer but you should test running 2 samples at the same time before running all of the scripts concurrently.  

---

### Answer by username_1 on 2024-06-01. Score: 3

The job array is a formal approach to solving this via SLURM
What I am not clear about is the number of files SRR8599992_1.fastq.gz SRR8599992_2.fastq.gaz I originally thought the final number of files was around 100, I've guessed at 120
#SBATCH --job-name=pangenome_spade
#SBATCH --account=lilianasalvador
#SBATCH --ntasks=120
#SBATCH --cpus-per-task=16 
#SBATCH --mem-per-cpu=5gb  
#SBATCH --time=02:00:00 
#SBATCH --array=599992-5600100

module load python/3.9  
module load spades/3.15.0-hpc 

spades.py -1 'SRR'${SLURM_ARRAY_TASK_ID}'_1.fastq.gz' -2 'SRR'${$SLURM_ARRAY_TASK_ID}'_2.fastq.gz' -o /groups/lilianasalvador/noah_100/'SPAdes_output_'${SLURM_ARRAY_TASK_ID}'.fastq' -t 16 --careful 

I would test this via an echo test script writing to a specific file to make sure the spades.py command produces the correct command line for each iteration.
Things to note ...

Array jobs is specified by the $SLURM_ARRAY_TASK_ID variable and #SBATCH --array=599992-5600100
ntasks is the total number of spades assemblies you are performing here I guessed at 120
You are parallelising via -t 16 therefore this must be reflected in your submission job via --cpus-per-task
The ouput file MUST use $SLURM_ARRAY_TASK_ID otherwise you will continually override your out file.


This is not an optimal SLURM job because ...

-t 16 you need to check with your HPC engineer, this might take a long time to get through the queue and an alternative architecture to permit 16 CPUs can get onto the cluster. HTC users will jump ahead of you, so on a hypervisor setup you can specify a separate cluster for your work. This is not well explained but hope it makes sense.
Shifting to high throughput computing (HTC) could be considered here to get this through a lot quicker, e.g. this is 120 sequential assemblies, 10 sequential assemblies for 12 separate submissions would be a lot quicker (you just alter the "array job" inputs range.


---

